Dealing with an extremely large dimension in Gemma 3 models.
TL;DR: While working with Gemma 3 models
Transformers. LLMs are typically decoder-only transformers
The residual stream. Conceptually, an LLM can be thought of as a multi-layer model that reads representations, then computes and applies updates to them, then reads the final representations to produce the output. In particular, a transformer block, which itself contains an attention layer and a feed-forward layer, reads from these embeddings, performs some black-box computation, then applies the change to the input embeddings via a residual connection. We call the output of a transformer block an activation vector $h_l \in \mathbb{R}^{d_{model}}$. The outputs of all layers are also called the “residual stream”
Interpretability. It turns out that the activation vectors are not mere numbers; they contain information about what the LLM has learnt, i.e. how models “represent” the input data. That is to say, if we read the activation vectors, we can identify geometric structures that correspond to human concepts. So, we also sometimes refer to activation vectors as “model representations”, especially if we can find specific vectors that seem to represent actual concepts.
Linear representations. The most popular realization of this idea is the linear representation hypothesis
Linear activation steering. Let’s say the model is processing some negative input $X^{neg}$, has some negative representation $h^{neg}$, and outputs some negative token $Y^{neg}$, but we want a positive output instead. We can directly intervene on $h$ by adding a steering vector $v^{neg \rightarrow pos}$ which points from negative to positive, so that $h^{neg} + v \rightarrow h^{pos}$. The later layers will then read from this representation and produce the positive output $Y^{pos}$
This activation steering technique is what I’m working with. To sum up, we can manipulate a model’s behavior by directly changing its activation vector, and we can do this by adding a steering vector. How the steering vector is computed is another topic, but typically it’s a difference-in-means over some dataset, as in Contrastive Activation Addition (CAA)
Representation analysis. Another thing we can do is measure how similarly an LLM represents two concepts. And we would reasonably expect model representations to reflect the real world. For example, if we take the representations for coffee and tea, we would expect their similarity to be higher than, say, between coffee and bread, but lower than between coffee and cà phê (the Vietnamese word for coffee). We typically measure similarity by cosine, which has a value between −1 and 1. The closer cosine is to 1, the more similar the representations; 0 means unrelated, and −1 means they point in opposite directions.
How I found them. In my project, I computed one steering vector per language (four languages in total) for the concept I’m studying. I wanted to know whether the model uses a similar direction for this concept across languages, so the first thing I did was compute the cosine similarity between each pair of language vectors.
For Gemma-3-4B and 12B, the mean pairwise cosine was around 0.96. I initially took this at face value and read it as evidence that the model uses a largely shared, language-independent representation for this concept.
Later, I asked ChatGPT to proofread my draft. Besides the language edits, it noted that 0.96 seemed high for four quite different languages, and asked whether something else could be driving the number. This was a reasonable question, especially since Qwen and Llama were at around 0.66–0.68.
So I looked at the individual dimensions of the vectors. In each Gemma model, one dimension was much larger than the rest. In Gemma-3-4B and 12B, it was over 2000$\times$ larger than the median dimension.
This phenomenon has been reported before, under names such as “outlier dimensions”, “rogue dimensions” and “massive activations”
Detecting them. The procedure is simple:
| Model | Ratio to median | Share of squared length |
|---|---|---|
| Gemma-3-1B | 525× | 0.2–52.7% (varies by language) |
| Gemma-3-4B | 2093× | 85.5–92.6% |
| Gemma-3-12B | 2198× | 83.9–93.2% |
| Qwen3-4B | 54× | 0.1–3.8% |
| Qwen3.5-9B | 166× | 0.0–1.3% |
| Llama-3.1-8B | 38× | 0.1–1.9% |
In Gemma-3-4B and 12B, this single dimension accounts for roughly 90% of the vector’s squared length. In Qwen and Llama, the top-ranked dimension accounts for at most a few percent.
Effect on cosine similarity. Recall that the cosine similarity between two vectors $a$ and $b$ is
\[\cos(a, b) = \frac{a \cdot b}{\|a\| \, \|b\|}\]We can split each vector into two parts: its value in the massive dimension ($M_a$ and $M_b$), and the remaining dimensions ($a’$ and $b’$). Then
\[\cos(a, b) = \frac{M_a M_b + a' \cdot b'}{\sqrt{M_a^2 + \|a'\|^2}\,\sqrt{M_b^2 + \|b'\|^2}}\]When $M_a$ and $M_b$ are both large and have the same sign, the $M$ terms dominate both the numerator and the denominator, and the cosine approaches 1. The term $a’ \cdot b’$, which reflects how the vectors differ in the remaining dimensions, contributes very little.
As a toy example, take $a = [1000, 1, 2]$ and $b = [1000, -2, 1]$. In the last two dimensions, these vectors are orthogonal (cosine 0). The full cosine, however, is above 0.99999.
The opposite can also happen. If $M_a$ is large but $M_b$ is small, $a$ points mostly along the massive dimension while $b$ does not, and the cosine is pulled down instead. This is the case for Gemma-3-1B, discussed below.
After zeroing the massive dimension, the mean pairwise cosine changed as follows:
After this correction, the Gemma models are in a similar range to Qwen and Llama. Most of the high similarity I first observed came from this one dimension.
Gemma-3-1B behaves differently. Its MAD is large in some languages and small in others (between 0.2% and 52.7% of the squared length), so it makes the language vectors look less similar. Zeroing it raises the mean cosine from 0.540 to 0.642.
This is consistent with earlier observations that a few dominant dimensions can strongly affect cosine similarity between representations
Effect on steering. A large magnitude does not necessarily mean the dimension matters for the model’s behavior. To check this, I zeroed the MAD in each steering vector, rescaled the vector to its original norm (so that the overall strength of the intervention stays the same), and re-ran the within-language and cross-language steering experiments.
The results barely changed. The steering success rate (how often steering flips the model’s output) changed by less than 1 percentage point on average, and by at most 4 points.
| Model | Mean change | Max change |
|---|---|---|
| Gemma-3-1B | 0.7 pp | 3 pp |
| Gemma-3-4B | 0.4 pp | 2 pp |
| Gemma-3-12B | 0.5 pp | 4 pp |
This suggests that the part of the steering vector responsible for the effect lies in the other dimensions, not in the MAD.
What I do now.
Open questions. I don’t yet know why these dimensions appear in Gemma but not in Qwen or Llama (at least at the layers I checked), or what role they play. Sun et al.