[Target audience: Me from a week ago, and people who have some understanding of ML but want to understand transformers better on a technical level.]

Free advice for people learning new skills: ask yourself random questions. In answering them, you’ll strengthen your understanding and find out what you really understand and what’s actually useful. And some day, if you ask yourself a question that no one has asked before, that’s a publication waiting to happen!

So as I was reading up on transformers, I got fixated on this question: where are the 175 billion parameters in the architecture? Not in the literal sense (the parameters are in the computer), but how are they “spent” between various parts of the architecture - the attention heads vs feed-forward networks, for instance. And how can one calculate the number of parameters from the architecture’s “size hyperparameters” like dimensionality and number of layers?

The goal of this post is to answer those questions, and make sense of this nice table from the GPT-3 paper, deriving the column from the other columns.

Most of Table 2.1 from the GPT-3 paper. How can we derive the number of parameters from the “size hyperparameters” (rightmost columns)?

Primary Sources

Lots of resources about transformers conjure information from thin air, and I want to avoid that, so I’m showing all my work here. These are the relevant parts of the sources^[1] we'll draw from:

Exhibit A: Equation 2 from the GPT-1 paper.

Exhibit B: Half of Figure 1 from the original transformer paper.

Exhibit C: An unnamed equation on page 5 in the original transformer paper. In the terminology of GPT, $d_{k}$ = $d_{v}$ = $d_{h e a d}$ , and $h = n_{h e a d s}$

Exhibit D: Equation 2 in the original transformer paper.

Three more details we’ll use, all from Section 2.1 of the GPT-3 paper:

The vocabulary size is $[n_{v o c a b} =] 50257$ tokens (via a reference to Section 2.3 of the GPT-2 paper)
The feed-forward networks are all a single layer which is “four times the size of the bottleneck layer”, so $d_{f f} = 4 d_{m o d e l}$
“All models use a context window of $n_{c t x} = 2048$ tokens.”

Variable abbreviations

I’ll use shorthand for the model size variables to increase legibility:

$\begin{matrix} n_{l a y e r s} & = & x d_{m o d e l} & = & y n_{h e a d s} & = & z d_{h e a d} & = & w n_{v o c a b} & = & v n_{c t x} & = & u \end{matrix}$

Where are the Parameters?

From Exhibit A, we can see that the original 1-hot encoding of tokens U is first converted to the initial “residual stream” $h_{0}$ , then passed through transformer blocks (shown in Exhibits B-D), with $n_{l a y e r s}$ blocks total^[2]. We'll break down parameter usage by stage:

Word Embedding Parameters
- $W_{e}$ is the word embedding matrix.
- Converts the shape ( $n_{c t x}$ , $n_{v o c a b}$ ) matrix $U$ into a $(n_{c t x}, d_{m o d e l})$ matrix, so $W_{e}$ has size $(n_{v o c a b}, d_{m o d e l})$ , resulting in $v y = n_{v o c a b} d_{m o d e l}$ parameters.
Position Embedding Parameters
- $W_{p}$ is the position embedding matrix. Unlike the original transformer paper, GPT learns its position embeddings.
- $W_{p}$ is the same size as the residual stream, $(n_{c t x}, d_{m o d e l})$ , resulting in uy = $n_{c t x} d_{m o d e l}$ parameters
Transformer Parameters - Attention
- The attention sublayer of the transformer is one half of the basic transformer block (Exhibit B). As shown in Exhibit C, each attention head in each layer is parameterized by 3 matrices, $W_{i}^{Q}, W_{i}^{K}, W_{i}^{V}$ , with one additional matrix $W^{O}$ ^[3] per layer which combines the attention heads.
- What Exhibit C calls $d_{k}$ and $d_{v}$ are both what GPT calls $d_{h e a d}$ , so $W_{i}^{Q}, W_{i}^{K},$ and $W_{i}^{V}$ are all size $(d_{m o d e l}, d_{h e a d})$ . Thus each attention head contributes $3 d_{m o d e l} d_{h e a d}$ parameters.
- What Exhibit C calls $h$ is what GPT calls $n_{h e a d s}$ , so $W^{O}$ is size $(n_{h e a d s} * d_{h e a d}, d_{m o d e l})$ and therefore contributes $n_{h e a d s} d_{h e a d} d_{m o d e l}$ parameters.
- Total parameters per layer: For a single layer, there are $n_{h e a d s}$ attention heads, so the $W_{i}^{Q}, W_{i}^{K}$ , and $W_{i}^{V}$ matrices contribute $3 d_{m o d e l} d_{h e a d} n_{h e a d s}$ parameters, plus an additional $n_{h e a d s} d_{h e a d} d_{m o d e l}$ parameters from $W^{O}$ , for a total of $4 d_{m o d e l} d_{h e a d} n_{h e a d s}$
- Total parameters: $4 x y z w = 4 d_{m o d e l} d_{h e a d} n_{h e a d s} n_{l a y e r s}$
Transformer Parameters - FFN
- The “feed-forward network” (FFN) is the other half of the basic transformer block (Exhibit B). Exhibit D shows that it consists of a linear transform parameterized by $W_{1}$ and $b_{1}$ , an activation function, and then another linear transform parameterized by $W_{2}$ and $b_{2}$ , as one might see in an MLP with one hidden layer^[4]. Note the remark in Exhibit D: “the linear transformations are the same across different positions, [but] use different parameters from layer to layer”, so we have one set of parameters $(W_{1}, W_{2}, b_{1}, b_{2})$ per layer.
- The matrices $W_{1}$ and $W_{2}$ are size $(d_{m o d e l}, d_{f f})$ and $(d_{f f}, d_{m o d e l})$ respectively. Since $d_{f f} = 4 d_{m o d e l}$ , each one contributes $4 d_{m o d e l}^{2}$ parameters.
- The bias terms $b_{1}$ and $b_{2}$ are size $d_{f f}$ $= 4 d_{m o d e l}$ and $d_{m o d e l}$ respectively, so they contribute $5 d_{m o d e l}$ parameters.
- Total parameters per layer: $8 d_{m o d e l}^{2} + 5 d_{m o d e l}$ parameters
- Total parameters: $8 x y^{2} + 5 x y = 8 d_{m o d e l}^{2} n_{l a y e r s} + 5 d_{m o d e l} n_{l a y e r s}$ parameters
Other Components
- The other components of the transformer architecture (such as the layer-norm in the “add and norm” block in Exhibit B) doesn’t use any parameters.

In total there are
$v y + u y + 4 x y z w + x y^{2} + 5 x y$

aka
$n_{v o c a b} d_{m o d e l} + n_{c t x} d_{m o d e l} + 4 d_{m o d e l} d_{h e a d} n_{h e a d s} n_{l a y e r s} + 8 d_{m o d e l}^{2} n_{l a y e r s} + 5 d_{m o d e l} n_{l a y e r s}$

parameters.

Is this correct?

Here’s a spreadsheet computing these values. Here’s how we did:

With one exception, that’s great! Our errors are ~1%! This takes me back to my college physics labs where the theory matches the “experiment” amazingly well. In the other tabs of the spreadsheet you can see we also do well at predicting the model sizes of the GPT-2 variants.

The one exception is GPT-3 XL, where we’re off significantly. We’ll discuss this more in the final section.

Analysis

Let’s look at some implications.

How many parameters are spent on each part of the architecture?

Now we can answer “how many parameters are used in the attention vs FFN parts of the architecture”":

We can see that:

Position embedding always take very few parameters.

Word embedding takes about 30% of the parameters for the smallest model, but a proportionally smaller amount as the model gets larger, ultimately <1% of parameters for the full-size GPT-3.

The remaining parameters are split 2:1 between the feed-forward network and the attention heads, except in GPT-3 XL, where there’s a 4:3 split. However, there’s nothing sacred about this 2:1 split. It’s a function of the ratio y:wz (which is 1:1 in all models except XL) and the FFN architecture (1 hidden layer with $d_{f f} = 4 d_{m o d e l}$ ), both conventions inherited from the original transformer paper.

How does model size depend on size hyperparameters?

For large models, the only terms that contribute significantly to the parameter count are the attention weights and non-bias FFN weights, which contribute $4 x y z w$ and $8 x y^{2}$ parameters, respectively (recall that $x = n_{l a y e r s}, y = d_{m o d e l}, z = n_{h e a d s}$ , and $w = d_{h e a d}$ ). For almost all GPT-3 sizes, y=zw, a convention adopted from the original transformer paper^[5], in which case the parameter count is O(xy^2). So parameter cost increases directly with the number of layers, but with the square of the internal dimensionality.

Guessing the size hyperparameters of GPT-4 from $n_{p a r a m s}$

Right now a hot trend on twitter is to hold your breath until you see a biblically accurate angel, count its eyes, and claim that GPT-4 will have that many parameters^[6].

Here at AIZI, we would never engage in such baseless speculation. Instead, let’s take that baseless speculation as factual and engage in a different baseless speculation: If we assume GPT-4 will have 100 trillion parameters and the same architecture as GPT-3, we can use numerical methods^[7] to find potential size hyperparameters. With $2^{k}$ layers and model size $2^{m}$ , you reach 100 trillion parameters when $k + 2 m = 43$ . With $k = 9$ and $m = 17$ , you have 512 layers and 131072-dimensional inner states, which has the benefit of sexy headline sizes increases over GPT-3: 500x parameters! 5x layers! 10x dimensionality!

Analysis of Transformers must include the FFNs

I love this work, where they analyze transformers that don’t have feed-forward networks, using them as the simplest example to build from. But knowing that two-thirds of GPT-3’s parameters are in the FFN sublayers means this approach alone is missing a lot of the picture. (And the amount its missing could be much more than 2/3rds of the picture: the FFN and attention sublayers are “mixing” information in different ways, so allowing both likely generates far more complexity than either one individually).

Open Questions

This project has given me a much better grasp of what’s going on in transformers, but there are questions I still haven’t answered yet:

Why was my prediction wrong about GPT-3 XL? You can fix the prediction by replacing the $4 x y z w$ term with $4 x y^{2}$ , which you can see on columns AA-AB of the spreadsheet, which works because $y = z w$ for all models except XL, where $1.5 y = z w$ . But I think the $4 x y z w$ term should be correct, in which case the GPT-3 table must have a typo in the number of parameters or the size hyperparameters. Am I wrong or is that a typo in the GPT-3 paper?
Why does GPT-3 use the same matrix for word embedding and final predictions? I would expect this to constrain the model, and the only potential upsides I can see are saving parameters (lol) and preserving interpretability (lmao)^[8]. Other resources like A Mathematical Framework for Transformer Circuits use different embedding/unembedding matrices - their $W_{E}$ and $W_{U}$ . Perhaps this is not necessary for GPT-3 since the final feed-forward network can perform an appropriate linear transformation, and in A Mathematical Framework they are looking at transformers without FFNs. But some properties (e.g. words being linear combinations of other words) cannot be changed by such a linear transformation, so having an entire new unembedding matrix could still add value.
The GPT-3 paper says “we use alternating dense and locally banded sparse attention patterns in the layers of the transformer”, citing this paper, and this sounds like it should affect the number of parameters! But our calculations never took this into account, and seem to be correct. What do they mean by sparse transformers? What do they contribute? Why am I able to compute the correct number of parameters by ignoring their “alternating dense and locally banded sparse attention patterns”?
GPT is called a “decoder only” architecture. Would “encoder only” be equally correct? From my reading of the original transformer paper, encoder and decoder blocks are the same except that decoder blocks attend to the final encoder block. Since GPT never attends to any previous block, if anything I feel like the correct term is “encoder only”.
Why not increase the context length and vocabulary size? The parameter costs of a larger context window and vocabulary size would be trivial. I could imagine that the vocabulary is already “good enough” and doesn’t need improving, but why not increase context length? Could this be the related to the “compute costs” I’ve heard rumors of?

If you know the answer to any of these questions, please leave an answer in the comments!

^{^}
1 Links to works cited, in order of appearance:
$∙$ Improving Language Understanding by Generative Pre-Training (“the GPT-1 paper” and yes I know GPT-1 is actually just GPT but I think its clearer to refer to it as GPT-1 in this context)
$∙$ Attention Is All You Need (“the original transformer paper”)
$∙$ Language Models are Few-Shot Learners (“the GPT-3 paper”)
$∙$ Language Models are Unsupervised Multitask Learners (“the GPT-2 paper”)
I also like these resources for learning about transformers:
$∙$ A Mathematical Framework for Transformer Circuits - An analysis of shallow transformers, highly recommended for including thorough detail missing in the papers.
$∙$ The Illustrated Transformer - An info post with lots of pictures.
$∙$ Building a ML Transformer in a Spreadsheet - A video an accompanying spreadsheet that implements a simple transformer (with hand-coded weights).
^{^}
In the final P(u) calculation the word embedding matrix $W_{e}$ is reused from step 1, so it doesn’t add more parameters. Although it’s possible to build a transformer with a different unembedding matrix, the parameter calculations come out correctly if you reuse these parameters.
^{^}
One could equivalently break $W^{O}$ into $n_{h e a d s}$ matrices $W_{i}^{O}$ , which is far more intuitive IMO, but I’ll try to keep the paper’s notation.
^{^}
The original transformer used RELU as its activation function, but the GPT-1 paper uses GELU, which I think is carried forward into GPT-3. Regardless, the activation function does not contribute any parameters.
^{^}
The original transformer explains that splitting attention between heads makes “the total computational cost… similar to that of single-head attention with full dimensionality.” Also, that’s enough dimensionality that the model can read the full dimensionality of the previous layer. So if, for a fixed model size y, computational and parameter costs are ~constant along the curve y=zw, what are you trading off by choosing larger or small z and w? The GPT-3 models choose different values for the w:z ratio, always with w>z, but otherwise varying the ratio. I also wonder if there’s reason to experiment with zw>y, to see if that leads to better performance (and OpenAI seems to have tried that, in GPT-3 XL, zw/y=1.5).
^{^}
I’m not going to link to all the ridiculous proclamations on twitter, but a popular guess is 100 trillion parameters. The strongest source I can find for that number is this interview with Andrew Feldman, but Andrew Feldman a) doesn’t work at OpenAI, b) was speaking in 2021 aka an eternity ago, and c) was giving a rough estimate.
^{^}
guess and check
^{^}
For those who are not terminally online, this translates to “those ships have sailed, this would increase parameters from 175B to 175.6B and decrease interpretability from 0 to 0”.