Feature of the Day: A Gemma SAE Atlas

A modern language model packs more concepts into every activation vector than it has dimensions, so the concepts sit on top of one another in superposition. A sparse autoencoder (SAE) is the tool that pulls them back apart: it learns an overcomplete dictionary of directions, each meant to stand for one human-readable concept, and re-expresses each activation as a sparse sum of a few of them. Decomposing dense activations into these interpretable features is one of the most active frontiers in mechanistic interpretability and AI safety.

This post starts a daily habit. Every entry takes a pretrained SAE from Google DeepMind's Gemma Scope suite, points it at Gemma 2 (2B), and lifts out one interpretable feature: what it fires on, how selective it is, and whether its direction is a real handle on the model. Day 1 is about standing up the pipeline and meeting the first feature. The whole thing runs end to end on a single free Kaggle T4, which is the point: interpretability you can reproduce on a laptop-class GPU.

What loads

I load Gemma 2 (2B) with no processing, so the residual stream stays in the raw, un-normalised form the SAE was trained on, then attach the canonical Gemma Scope SAE for layer 12 at width 16k. That SAE has 16,384 features and reads the post-block residual stream. Everything runs in float16. The canonical checkpoint resolves to the variant trained to an average L0 near 82, meaning it was tuned to fire roughly 82 features per token on its training distribution.

The probe is a varied set of 40 prompts: prose, code, dialogue, factual claims, instructions, and questions. A diverse corpus is what lets a genuinely interpretable feature stand out from prompt-specific noise. For each prompt I cache the layer 12 residual stream, encode it with the SAE, and stack the per-token feature activations into one tensor of shape (40, 21, 16,384): 40 prompts, up to 21 tokens each, 16,384 features.

Is the SAE healthy?

Before trusting any single feature, measure whether the dictionary behaves like a healthy sparse one. I mask out the BOS token and any padding, keeping only genuine content tokens, then count how many features fire per token, how often each feature fires, and how many never fire at all.

Across 486 content tokens the SAE fired a mean of 74.3 features per token (median 72, max 194), close to the checkpoint's nominal L0 of 82. Mean feature density, over the features that fired at all, was about 0.93 percent. And 8,379 of the 16,384 features (51.1 percent) never activated on this corpus, leaving 8,005 alive. That last number is the expected shape, not a defect: a 40-prompt probe cannot exercise a 16k-feature dictionary, so half of it staying dormant is exactly the sparsity these autoencoders are trained to produce. Dormant here means unexercised by this small corpus, not a dead unit that never fires on anything.

The strongest directions

Ranking features by their single strongest activation is a deliberate choice. A density ranking, how often a feature fires, surfaces broad, always-on features. A peak ranking surfaces the sharp, selective ones that fire hard on a specific token and stay quiet otherwise, which are the features most likely to carry a single clean meaning. Here are the top ten by peak activation over content tokens, straight from the notebook:

rank   feat     peak   density   top token          context
   1   4667    393.0    8.23%    'Therefore'        prompt 5,  pos 1
   2  11087    200.0    8.23%    'According'        prompt 22, pos 1
   3   1178    133.1    8.23%    'A'                prompt 23, pos 1
   4   6810     80.4   33.54%    ' approximately'   prompt 27, pos 6
   5   6889     63.8    8.23%    'import'           prompt 11, pos 1
   6  12971     60.4    0.41%    ' __'              prompt 24, pos 7
   7   9814     59.9    0.21%    ' as'              prompt 11, pos 3
   8   3781     56.7    0.62%    ' human'           prompt 39, pos 2
   9   1041     54.4   47.74%    ' autumn'          prompt 12, pos 5
  10   3379     54.2    2.06%    ' range'           prompt 17, pos 4

The same ten features, plotted by how hard they peak against how often they fire, tell the story better than the table:

Three families fall out. Four features stack on the dashed 8.23 percent line: 4667, 11087, 1178, and 6889. A density of 8.23 percent is 40 of 486 content tokens, and with 40 prompts that is about one firing per prompt, each on the very first content token. To the right sit the broad features. Feature 1041 fires on 47.74 percent of all tokens and 6810 on 33.54 percent; their listed top token (' autumn', ' approximately') is just where they happened to peak, not a clean meaning, which is exactly what a high-density feature looks like. To the left sit the rare, low-density features (' as', ' __', ' human' at 0.21 to 0.62 percent). The winner sits alone at the top: feature 4667 peaks at 393.0, nearly double the runner-up, at a moderate density. Selective and sharp at once, so it becomes the Feature of the Day.

What feature 4667 is

Feature 4667 fires on sentence-initial words and discourse markers. Its single strongest content token was 'Therefore' at position 1, and across the corpus it consistently lights up on the first content token (The, In, She, def, Therefore, According, A) while every other position stays near zero. The other three features on the 8.23 percent line behave the same way: sentence starters that fire once per prompt. That clean once-per-prompt pattern is only visible because position 0, the BOS token, was masked out first. Leave it in and the strongest direction is the attention-sink artifact that BOS attracts, not a real feature.

The auto-interpretations on Neuronpedia agree with the corpus evidence: looked up under the gemma-2-2b Gemma Scope release, layer 12, width 16k, feature 4667 reads as sentence-initial discourse markers that signal a logical or rhetorical transition, contrast, cause, or conclusion. A well-corroborated, genuinely interpretable direction.

Steering: is the direction real?

A feature is only interesting if its direction is a usable handle on the model rather than a coincidence. I take the feature's decoder vector, the direction it writes into the residual stream, normalise it, and add it back at layer 12 during generation at a few strengths. Same prompt, temperature 0.7, sampling on:

The claim here is modest and I want to keep it that way. The text stays fluent and coherent as the direction is pushed from 0 to 120, which is a real sanity check: the decoder direction is a usable handle on the model, not noise that shreds the output. It is not, on its own, proof of what the feature means. A clean causal demonstration for a sentence-initial feature wants a setup that can actually start new sentences, and that is a better fit for a later entry than for Day 1.

Where this is heading

The pipeline updates every day by changing the probe: a fresh prompt set, a different layer from 0 through 25, or a different SAE width. Re-run, and a new Feature of the Day falls out automatically, documented and added to the atlas. The daily cadence keeps the tooling sharp while building toward the flagship arc: training SAEs on the newest Gemma models on a local GPU, where no public SAEs exist yet. Each daily entry is a rehearsal for that larger contribution.

Day 1 of the atlas. Run it yourself: the notebook on Kaggle runs end to end on a single free Kaggle T4. Look up any feature yourself on Neuronpedia under the gemma-2-2b Gemma Scope release. References: Gemma Scope, arXiv 2408.05147; the pipeline is built on SAELens and TransformerLens.