● WRITING · 4 NOTES · PROSE AND EVIDENCE, SAME PIPELINE
Notes & write-ups
Write-ups from the bench — what worked, what collapsed under a corrected protocol, and the gates that caught it. The charts inside each post render from the same pinned results files as everything else on this site.
12 JUL 2026Feature of the Day: A Gemma SAE AtlasA daily tour through the interpretable features hiding inside Gemma 2 (2B). Day 1 builds the pipeline: a Gemma Scope sparse autoencoder with 16,384 features reads layer 12, fires a median of 72 of them per token, and leaves 51 percent dormant on a 40-prompt probe. Its sharpest direction, feature 4667 (peak activation 393), is a sentence-initial discourse-marker detector that lights up on 'Therefore', 'According' and 'A' and stays near zero everywhere else.interpretabilitysparse-autoencodersgemmagemma-scopemechanistic-interpretabilitykaggle08 JUL 2026Killing the 10 GB TensorMy 596M-parameter model's single biggest training tensor was a 9.96 GB fp32 logit matrix that exists only to be reduced to one scalar. On a free Kaggle T4, fused cross-entropy deletes it: the naive path OOMs past 4,096 tokens while Liger holds flat at ~3.2 GB out to 32,768, agreeing with the fp32 reference to ~1e-7.cudamemorycross-entropyligerkaggleqwen3-0.6b07 JUL 2026The Win That Was NotMy SFT ablation produced a +0.68-perplexity win with a confidence interval tight enough to look unimpeachable. It was an artifact of scoring the two arms on different token sets, and it collapsed to +0.009 on a fixed held-out set.qwen3-0.6bsftevaluationreproduce-then-attribute07 JUL 2026What the Gates CaughtSeven ways my own study tried to fool me (a token-soup data loader, an eval-token confound, a misread smoke log, a conflated data ratio) and the mechanical checks that caught each one before it became a claim.qwen3-0.6bmethodologypre-registrationreproduce-then-attribute
Numbers in prose are rounded; the charts are not./research has every number →