The problem
"I built a language model from scratch" is usually an unfalsifiable claim: the code runs, the loss goes down, and nothing proves the implementation matches the model it claims to reproduce. BuildFromScratch starts from the opposite premise — a reproduction is only real when it is bit-exact, and every claim built on top of it must carry the statistics to survive scrutiny.
Architecture
Two reproductions, each written single-file from a blank editor: SmolLM2-135M (134,515,008 parameters) and Qwen3-0.6B-Base (596,049,920 parameters). Each ships a verify.py parity gate that loads the official HuggingFace weights into the from-scratch class and compares logits in fp32 on a recorded probe:
That is the maximum absolute logit difference against the reference — zero, not "close". The gate is the non-negotiable test before any training run.
Reproduce, then attribute
The bit-exact build is the floor, not the finish line. From it, the project runs controlled single-variable studies across the lifecycle, with cross-run claims held to 3 seeds, iso-FLOP matching via an explicit FLOP-per-token accounting module, and a held-out noise floor. Three builds trained on identical 1.19B-token budgets — faithful baseline, modernized IMU-1 bundle, exploratory partial-RoPE:
A de-confound then attributes the bundle's win — on the canonical metric, only the architecture axis survives; the WSD schedule and z-loss contribute nothing significant:
Drilling into that axis, the IMU-1 architecture modules (value-residual, layernorm-scaling, head-gating) are individually significant on canonical bits-per-byte:
The largest effect in the whole study is data composition — switching pretraining data to dclm-edu moves code BPB by:
+0.7034✓What the evidence does not support
The three-build comparison is single-seed and in-distribution, so it is tagged descriptive and never rendered as a headline. Response-masked SFT did not separate from an iso-FLOP continued-pretraining control — a naive in-loop metric had shown a 0.68-PPL "win" that collapsed to roughly +0.01 once both arms were scored on one fixed held-out set. A pre-registered GRPO run at 0.6B beat neither the SFT floor nor a random-reward gate. All three nulls are published with the same prominence as the wins.
Every number above is fetched at render time from result files pinned to a commit in the repo; click any stat to see its source.