DWG NO. YB-PAPER-01 · PREPRINT · JULY 2026 · THE PAPER

Reproduce,
then attribute.

A Controlled Study of the LLM Training Lifecycle on a Bit-Exact Qwen3-0.6B Reproduction

YASH BISHNOI · EDINBURGH, UK · github.com/yashb98/BuildFromScratch
ABSTRACT — ADAPTED FROM THE PREPRINT, TRIMMED

Published improvements to language-model training rarely arrive with attribution: gains are reported for bundles of changes, at loosely matched compute, often from single runs — a reader learns that a recipe won without learning why. This paper carries a bit-exact reproduction of Qwen3-0.6B (fp32 max |Δlogits| = 0.0 against the reference on the recorded probe) through the full training lifecycle — pre-training, data composition, mid-training, supervised fine-tuning, and reinforcement learning with verifiable rewards — on a single GB10 machine, admitting a comparison only when exactly one variable differs, training FLOPs match within 5%, and the effect is a paired three-seed bits-per-byte delta with a 95% confidence interval on two corpora. Under these gates, a modernization bundle’s single-run, in-loop −17.9% validation-perplexity headline decomposes into three individually significant architecture effects plus an early-training optimizer effect, while its schedule and z-loss axes contribute nothing measurable; and a premium-data anneal improves code BPB by +0.2716 (95% CI [+0.2641, +0.2792], 3 seeds, paired) over an iso-token control that absorbs the learning-rate confound. Three nulls are reported at the same standard: SFT response-masking does not separate from its iso-FLOP control; GRPO at 0.6B (single seed, directional by design) confirms a pre-registered null against a random-reward gate; and context extension is gated off by a step-0 diagnostic. The same gates caught the project’s own errors — including an in-loop 0.68-perplexity “win” that collapsed to +0.009 on a fixed held-out set.

DOWNLOAD PDF ↓view repo ↗
pinned paper-release commit 94de6ef882…
THE EVIDENCE STANDARD · §3, EVIDENCE-GATED METHODOLOGY

One standard for every claim — wins, nulls, and the project’s own errors.

3 seeds, paired

Both arms train the identical seed set {0, 1, 2}; the effect is a paired difference of arm means, never a single lucky run.

iso-FLOP

Two arms compare only if exactly one variable differs and training FLOPs match within 5% — token-matched is not FLOP-matched, so the check is computed, not assumed.

Welch-t 95% CI

Student-t interval with Welch–Satterthwaite degrees of freedom; a delta is significant only if the entire interval lies on the improving side of zero.

pre-registration

The RLVR cohort's success gates were written down before training; the null is reported against those gates, not ones chosen after the fact.

evidence classes

Every number carries its class — headline (3-seed, CI-backed), descriptive (n=1, in-loop), or honest null — and no number is ever promoted above its class.

THE FIGURES · EACH ONE INTERACTIVE · EACH ONE WIRED TO A RESULTS FILE AT THE PINNED COMMIT
FIG 1— Parity gate replay · the gate everything rests on
verify_run.py — recorded replayDETERMINISTIC · BIT-EXACT✓ VERIFIED
$ press “run verify.py” to replay the recorded parity gate
Replay lines reconstructed deterministically from verify_run.py print statements + the committed results/verify.json values; per-stage timings omitted (never recorded; total wall time 30.0s). Every number is the recorded one; the output path is shown repo-relative (the script prints it absolute).
§5 · Bit-Exact Reproduction and the Faithful Baseline — verify_run.py replay, reconstructed from the committed verify.json: an fp32 forward pass on the recorded probe against the HuggingFace reference, with argmax agreement. Every comparison downstream of this page assumes this gate passed first — pytest before training, parity before claims.
FIG 2— Gap to the original
Gap to the original — val PPL vs training tokensDESCRIPTIVE · n=1✓ VERIFIED
hover, tap, or focus + ← → to inspect a point · Enter opens provenance
Validation perplexity versus training tokens, original checkpoint and reproductions
checkpointroletraining tokensval PPLgap pairing
Smoke proberepro65.5M95.87—
Phase-A best (lr 2.4e-3)repro131M46.313.5x · ~275,000x less data
Phase-B faithful (2TPP)repro1.19B28.652.14x · ~30,000x less data
Original Qwen3-0.6B-Baseoriginal36T13.4—
notes & provenance

metric: val PPL (held-out FineWeb-Edu, 204,800 tok, 50x4096 windows)

Single-run, in-loop comparison (n=1, descriptive). The ratio pairs are load-bearing and must never be conflated: 2.14x at ~30,000x less data (1.19B vs 36T tokens) and 3.5x at ~275,000x less data (131M vs 36T). Computed: 28.65/13.4 = 2.1381; 46.31/13.4 = 3.4560.

  • Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/results/original_vs_repro.txt
  • Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/results/qwen3_baseline2tpp_after.txt
  • Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/results/qwen3_after.txt
exported result file ↗
§5 · Bit-Exact Reproduction and the Faithful Baseline — Validation perplexity versus training tokens on the identical FineWeb-Edu validation slice, identical eval code: the original Qwen3-0.6B-Base against the faithful baseline and the learning-rate-sweep arm. Single run per point, in-loop — descriptive by the paper's own caps, and never conflated with the suite-stamped numbers.
FIG 3— De-confounding the IMU-1 bundle

Deconfound Explorer

P1 AXIS3-SEED · 95% CI✓ VERIFIEDP2 SUB-DRILL3-SEED · 95% CI✓ VERIFIED

Toggle a training change on or off; the chart stacks its measured contribution into a predicted ΔBPB (the sum assumes additivity). Drill into the arch bundle to see how good that assumption really is.

arch bundle+0.1179wsd schedule · n.s.+0.0245z-loss · n.s.-0.0034predicted ΔBPB = +0.1390sum of toggled effects · assumes additivity · +0.0211 of it is n.s. (noise-level)0.000.050.100.15ΔBPB vs faithful baseline (wikitext-2) · + = better
§6 · Pre-Training: From a Confounded Headline to Attributed Causes — The modernization bundle's single-run in-loop headline, decomposed axis by axis at iso-FLOP: schedule, regularizer, and architecture in Phase 1; the architecture bundle sub-drilled into value-residual, layernorm-scaling, and head-gating in Phase 2. Three seeds per arm, paired 95% CIs, suite text-lm-v2 — toggle the axes to see what actually carried the win.
FIG 4— Data composition & the premium-data anneal
dclm-edu vs 50/50 mix — ΔBPB over FineWeb-Edu control (code_py)3-SEED · 95% CI✓ VERIFIEDΔBPB vs FineWeb-Edu baseline (code_py)↑ higher is better
  dclm-edu (100%)0.7034■ best
  50/50 mix0.5904
select a row (Enter/Space or click) for per-seed detail
BuildFromScratch/Qwen3-0.6B/results/mix.json @ fe18745556 · scripts/export_site_results.pyraw ↗
PREMIUM-MIX ANNEAL · CODE BPB VS ISO-TOKEN CONTROL
+0.2716 BPB✓
3-SEED · 95% CI✓ VERIFIED

The cooldown alone is not the effect — the iso-token control runs the identical schedule on ordinary data, so what remains is the data.

§7 · Data Composition — §8 · Mid-Training — Left: code-BPB improvement over the FineWeb-Edu control for the pure dclm-edu arm and the 50/50 mix, paired across-seed 95% CIs, three seeds per arm. Right: the mid-training anneal — a 1-sqrt cooldown on the premium mix against an iso-token control that absorbs the learning-rate confound. The anneal effect is the paper's cleanest mid-training number; hover it for the source file.
FIG 5— Effective-context-length passkey ladder
Passkey ladder — retrieval accuracy vs context rung3-SEED · 95% CI✓ VERIFIED
hover, tap, or focus + ← → to inspect a rung
Passkey retrieval accuracy by rung and arm, Wilson 95% CI, probe count n
rungbasefineweb annealmix anneal
5120.200 [0.105, 0.348], n=400.308 [0.233, 0.396], n=1200.817 [0.738, 0.876], n=120
10240.200 [0.105, 0.348], n=400.200 [0.138, 0.280], n=1200.400 [0.317, 0.489], n=120
20480.200 [0.105, 0.348], n=400.200 [0.138, 0.280], n=1200.225 [0.159, 0.308], n=120
40960.025 [0.004, 0.129], n=400.400 [0.317, 0.489], n=1200.400 [0.317, 0.489], n=120
61440.700 [0.546, 0.819], n=400.700 [0.613, 0.775], n=1200.708 [0.622, 0.782], n=120
81920.350 [0.221, 0.505], n=400.408 [0.325, 0.498], n=1200.450 [0.364, 0.539], n=120
notes & provenance

Long-context passkey ladder (ecl-ladder-v1), rungs 512-8192, 5 depths x 8 keys = 40 probes per rung per cell; anneal arms pooled over 3 seeds (n=120/rung), base is a single checkpoint (n=40/rung). Wilson 95% CIs. The 4096 dip is BASE-ONLY (0.025 at its own trained length; both annealed arms hold 0.400 there) and unexplained — see paper §13. All three lines bump at 6144. Per-seed detail in ecl_ladder.json.

  • Qwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/verdict.json
  • Qwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/ecl_ladder.json
exported result file ↗
§8 · Mid-Training: Premium-Data Annealing and the Context Gate — Passkey accuracy by context rung, 512–8192 tokens (suite ecl-ladder-v1): the un-annealed base against the FineWeb-Edu-anneal control and the mix-anneal treatment, 40 probes per rung per cell, anneal arms pooled over three seeds, Wilson 95% CIs. Both anneal arms repair the base's anomalous dip at its own trained length; the mechanism is unexplained and flagged as an open question in §13.
FIG 6— The SFT eval-token confound — honest null nº 1
SFT response-masking vs iso-FLOP control — ΔPPL (positive = SFT better)HONEST NULL✓ VERIFIEDΔPPL, SFT vs control (positive = SFT better)↑ higher is better
  in-loop 'win' (confounded)0.6787
  held-out, response-masked (corrected)0.0092
  held-out, full-sequence (cross-check)-0.0064n.s.
select a row (Enter/Space or click) for per-seed detail
BuildFromScratch/Qwen3-0.6B/results/sft_null.json @ fe18745556 · scripts/export_site_results.pyraw ↗
§9 · Supervised Fine-Tuning: The Win That Was Not — The in-loop comparison scored the SFT arm on response tokens and the control on all tokens — an incomparable pair that produced a 0.68-perplexity apparent win. Re-scored on one fixed held-out response-masked set the gap all but vanishes, and the full-sequence cross-check flips its sign. Directional by the paper's own verdict caps, reported at the same prominence as the wins.
FIG 7— RLVR at 0.6B — the pre-registered null
GRPO vs SFT floor vs random-reward gate vs RFT control — pass@1, GSM8KHONEST NULL✓ VERIFIEDpass@1, gsm8k (n=100 items, T=0.8, 8 samples)↑ higher is better
n=1 / DIRECTIONAL — 4 of 4 entries are single-run (no seeds); read as directional evidence, not an attributed effect
  SFT floor0.0112
  GRPO0.01
  random-reward gate0.01
  RFT (iso-generation control)0.015
select a row (Enter/Space or click) for per-seed detail
BuildFromScratch/Qwen3-0.6B/results/grpo_null.json @ fe18745556 · scripts/export_site_results.pyraw ↗
GRPO reward, step by step
fraction verifier-correct per step · 300 steps
N=1 · SINGLE SEEDHONEST NULL✓ VERIFIED
20 steps
00.020.04150100150200250300GRPO step0.50random-reward arm ≈0.50flat at ≈0.9% — pre-registered null confirmed
§10 · RLVR at 0.6B: A Pre-Registered Null — Top: pass@1 on GSM8K (Wilson 95% CIs where recorded — SFT floor and Dr.GRPO) for the SFT floor, Dr.GRPO, the random-reward gate arm, and the iso-generation-compute RFT control — all intervals overlap and both pre-registered gates fail. Bottom: GRPO's per-step training reward over 300 steps, flat with no trend. Single training seed by pre-registered design, so the verdict is capped at directional — a null, reported as one.
EVIDENCE MANIFEST · 61 CLAIMS · 6 CLUSTERS · BUILT 2026-07-06

Every paper number, traced to its source file.

The manifest below is the paper’s own audit trail — each claim names the file that proves it, linked at the pinned paper-release commit. 2 of 61 entries are recorded as contradicted — citation errors the audit itself caught. They stay visible; that is the point.

✓ fetched from the pinned commit · raw JSON ↗

claimvaluesource filestatus
Reproduction & verification · repro-verify · 10 claims
The Qwen3-0.6B reproduction is verified bit-exact against HF weights (max |Δlogits| = 0.0, argmax agreement)."max_abs_error": 0.0, "relative_error": 0.0, "argmax_match": true, "passed": true (dtype float32, tolerance 0.001, prompt "The capital of France is", hf_next_token_id 12095 = our_next_token_id 12095, token " Paris")Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/results/verify.jsonfound
The reproduction's parameter count is 596,049,920.596,049,920 (architecture_plan.md forward-calc table row '| **Total** | **596,049,920** |', line 60; test_model.py line 79 'expected = 596_049_920' asserted within 1%)Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/architecture_plan.mdfound
The faithful base was trained on ~1.19B FineWeb-Edu tokens at sequence length 4096.token_budget=1,189,478,400; args seq_len: 4096, micro_batch: 4, grad_accum: 4, steps: 18150, tok/step=65,536, peak_lr 0.0024, dtype bfloat16, seed 0; data line: 'Streaming FineWeb-Edu sample-10BT...' then 'streamed 1,191,478,748 train + 300,012 val tokens'Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/results/qwen3_baseline2tpp_train.logfound
The faithful base lands within 2.14x of the original Qwen3-0.6B PPL using ~275,000x less data.2.14x half IS file-backed: 28.65 (qwen3_baseline2tpp_after.txt line 2 'val PPL: 185810.49 -> 28.65') / 13.400 (original_vs_repro.txt line 2 'ORIGINAL Qwen3-0.6B-Base (36T tok) val PPL = 13.400') = 2.138; build README line 165 '| Gap vs original (13.40) | **2.14×** |'. BUT ~275,000x belongs to a DIFFERENT run: original_vs_repro.txt line 7 pairs '~275,000x less data (36T vs 131M)' with the 131M-token Phase A best repro (lr24 46.310, gap 3.5x), and the build README line 189 says of the 2.14x run: 'The token budget is ~30,000× smaller than the real run; a 2.14× PPL gap at 1.19B tokens is the expected, honest outcome.'Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/results/original_vs_repro.txtcontradicted
Baseline training throughput is ~7,300 tok/s on the GB10.'Current Qwen3 PyTorch build: **~7,300 tok/s** with `torch.compile`, micro_batch=4 @ seq 4096 (measured).' (jax_vs_pytorch_tradeoffs.md line 43). Underlying artifacts: throughput_probe.json compiled pass tokens_per_second = 7167.4 (uncompiled 3787.5, speedup 1.89, peak_mem_gb 52.4); Phase B train log sustained 7,444-7,482 tok/s (final line: 'step 18150/18150 ... tok/s 7,444'); Phase A lr30 log ~7,272-7,316 tok/s; build README line 167 quotes '~7,480 tok/s'jax_vs_pytorch_tradeoffs.mdfound
Faithful-baseline eval numbers with suite_version: final FineWeb-Edu val PPL 28.65; eval-harness suite PPLs wikitext2 37.01 and code 438.67 under text-lm-v2.In-loop final val PPL: 'val PPL: 185810.49 -> 28.65' (qwen3_baseline2tpp_after.txt line 2 — NO suite_version; train-loop eval on FineWeb-Edu val, 204,800 tok). Suite-stamped numbers: suite_version 'text-lm-v2', date 2026-06-16 22:26:53, target_ckpt checkpoint_qwen3_baseline2tpp.pt — ppl.wikitext2_val.target = 37.010055463333096 (corpus wikitext2_raw_v1_val, n_tokens 204600); ppl.code_py.target = 438.67295146042875 (corpus codeparrot_clean_valid, n_tokens 204600); noise_floor.wikitext2_val.floor_abs = 1.3039039020038814 (floor_pct 3.7305), noise_floor.code_py.floor_abs = 280.56392586197546 (floor_pct 68.2455)Qwen3-0.6B/experiments/2026-06-16_qwen3-0.6b_eval-faithful/eval/suite_results.jsonfound
The SmolLM2 reproduction has exactly 134,515,008 parameters.'params: 134,515,008 (target 134,515,008)' + 'tied: True' (param_count.log lines 1-2); summary.json: "Param count (unique)": "134,515,008 (target 134,515,008 ✓)"SmolLM2-134(base)/results/param_count.logfound
The SmolLM2 reproduction is verified bit-exact vs HuggingFaceTB/SmolLM2-135M (max |Δlogits| = 0.0).'max |Δlogits| = 0.000e+00' / 'relative = 0.000e+00' / HF next token ' the' = ours / '✓ Architecture parity verified.' (parity.log lines 6-11); comparison_with_hf.md: final-logits parity CPU max|Δ| = 0.00e+00, per-layer (30 layers) CPU max|Δ| = 0.00e+00 everywhere, greedy generation 5/5 exact; summary.json "max |Δlogits| vs HF": "0.000e+00"SmolLM2-134(base)/results/parity.logfound
(Adjacent) Original Qwen3-0.6B-Base scores val PPL 13.400 on the identical 300k-token FineWeb-Edu val slice; best 131M-token Phase A repro (lr24) = 46.310, a 3.5x gap.'ORIGINAL Qwen3-0.6B-Base (36T tok) val PPL = 13.400 (204,800 tok, 21s)'; 'REPRO lr24 (131M tok, from scratch) val PPL = 46.310 gap x 3.5' (also lr17 46.892 x3.5, lr30 49.276 x3.7)Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/results/original_vs_repro.txtfound
(Adjacent) Mid-training PPL descent and run stats of the faithful 2 TPP run.eval @ 18000: val PPL=28.66; final AFTER val PPL=28.65 (BEFORE 185810.49); 'Training complete in 2663.1 min.'; mem 52.4GB flat throughout; build README line 167 gives the full descent 60.10 (@2k) -> ... -> 28.65 (final)Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/results/qwen3_baseline2tpp_train.logfound
IMU-1 de-confound (pre-training) · imu1-deconfound · 9 claims
The modernized IMU-1 bundle beats the faithful baseline by 17.9% at matched compute (final val PPL 23.52 vs 28.65, both 18,150 steps / ~1.19B tokens / 2 tokens-per-param on FineWeb-Edu with an identical held-out val set).faithful 28.65 (final) · modernized (IMU-1 NorMuon) 23.52 (final); 17.9% stated as 'IMU-1 bundle beat the faithful baseline by 17.9%' and arithmetically (28.65-23.52)/28.65 = 17.91%Qwen3-0.6B/builds/comparison/README.mdfound
The exploratory partial-RoPE 0.25 build lost to the faithful baseline (final val PPL 29.54 vs 28.65).29.54 ('DONE final val PPL=29.54'); README table: 'exploratory partial-RoPE 0.25 | 29.54 | final'; ranking modernized < faithful < partial-RoPE 0.25 < partial-RoPE 0.10 (50.71, died incomplete at step 5450/18150)Qwen3-0.6B/builds/2026-06-08_reproduce-exploratory_qwen3-0.6b/results/qwen3_prope25_2tpp_train.logfound
NorMuon-vs-AdamW single-variable iso-FLOP ablation (42M tokens, 3 seeds/arm): NorMuon on the 2D hidden weights improves BPB by +0.4743 on wikitext-2 (95% CI [+0.4435, +0.5052]) and +0.5016 on code (95% CI [+0.4560, +0.5471]), significant; verdict WIN (scoped as an early-training optimization-speed signal, not a scale claim).wikitext2_val: adamw_mean 2.1098171365956357 vs normuon_mean 1.6354916346714725, improvement_bpb 0.47432550192416323, ci95 [0.4434844613250229, 0.5051665425233036], significant true, df 3.8610499345338067, n [3,3]; code_py: adamw_mean 3.3846985755955523 vs normuon_mean 2.8831399238053073, improvement_bpb 0.5015586517902451, ci95 [0.4559911731303807, 0.5471261304501094], significant true, df 3.988970056422409, n [3,3]; verdict "win"Qwen3-0.6B/experiments/2026-06-16_qwen3_normuon-vs-adamw/results/verdict.jsonfound
The matched-config AdamW LR sweep shows AdamW is flat within seed noise across 1.7e-3–3.5e-3 and no AdamW LR comes within ~10x of closing the NorMuon gap (full LR spread ~0.047 bpb vs the +0.474 gap), so the result is 'NorMuon vs AdamW-anywhere-in-a-reasonable-LR-range', not an undertuned baseline.lr_sweep_bpb.json: 1.7e-3 wikitext_bpb 2.124645242917025 / code_bpb 3.36670692543473; 3.5e-3 wikitext_bpb 2.1223452336564987 / code_bpb 3.484039905578447; 4.8e-3 wikitext_bpb 2.156916637970004 / code_bpb 3.457733614887074; 2.4e-3 point = the 3-seed cohort mean 2.1098 (verdict.json). RESULT.md §4.1: '1.7e-3 → 2.1246, 2.4e-3 → 2.1098, 3.5e-3 → 2.1223, 4.8e-3 → 2.1569 ... full AdamW LR spread is ~0.047 bpb, ~10× smaller than NorMuon's +0.474 advantage'Qwen3-0.6B/experiments/2026-06-16_qwen3_normuon-vs-adamw/results/lr_sweep_bpb.jsonfound
De-confound Phase 1 (axes wsd/zloss/arch vs the faithful AdamW baseline, 3 seeds/arm, 2000-step proxy): only 'arch' is a significant driver — wikitext BPB +0.1179 CI[+0.1005, +0.1354] and code +0.3049 CI[+0.2588, +0.3511] — while wsd and zloss are not significant on either corpus; overall_verdict 'attributed' with drivers=[arch].arch: wikitext2_val improvement_bpb 0.11792615800601025 ci95 [0.10045055390862463, 0.13540176210339586] significant=true; code_py improvement_bpb 0.3049240450045283 ci95 [0.2587722809820044, 0.3510758090270522] significant=true. wsd: wikitext +0.02445100663943567 ci95 [-0.017262911764197773, +0.06616492504306912] significant=false; code +0.03613565282082609 ci95 [-0.06492886457337839, +0.13720017021503056] significant=false. zloss: wikitext -0.003374986704193006 ci95 [-0.017536542115922118, +0.010786568707536106] significant=false; code -0.0007094693010314401 ci95 [-0.052240000161460506, +0.050821061559397626] significant=false. drivers ["arch"]; overall_verdict "attributed"Qwen3-0.6B/experiments/2026-06-18_qwen3-0.6b_imu1-deconfound-p1/verdict.jsonfound
Phase 2 arch sub-drill: each of the three IMU-1 arch flags — value-residual (vr), layernorm-scaling (ln), head-gating (hg) — is individually significant on BPB on BOTH corpora vs the reused Phase-1 baseline; drivers=[vr,ln,hg], overall_verdict 'attributed'.vr: wikitext improvement_bpb 0.035502910568182555 ci95 [0.011595720478729928, 0.05941010065763518] sig=true; code 0.10715460338422833 ci95 [0.0492464984813345, 0.16506270828712216] sig=true. ln: wikitext 0.03367590105393914 ci95 [0.017509206440970627, 0.04984259566690765] sig=true; code 0.06055088208493942 ci95 [0.01983113346350228, 0.10127063070637655] sig=true. hg: wikitext 0.025562479271331817 ci95 [0.01251982603769344, 0.03860513250497019] sig=true; code 0.04215309989332772 ci95 [0.00805499069515924, 0.07625120909149619] sig=true. drivers ["vr","ln","hg"]; overall_verdict "attributed"Qwen3-0.6B/experiments/2026-06-21_qwen3-0.6b_arch-subdrill-p2/verdict.jsonfound
All cross-run comparable numbers in this cluster carry the eval-harness suite_version stamp 'text-lm-v2' and n=3 seeds per arm.suite_version = "text-lm-v2" stamped in: normuon results/verdict.json (+ n:[3,3] per corpus) and results/cohort_bpb.json; p1 verdict.json + cohort_bpb.json; p2 verdict.json + cohort_bpb.json; and the three build-checkpoint suite_results.json files (eval-faithful/-modernized/-prope25). Seeds: 3 per arm everywhere (3-element seed BPB arrays; c5_evidence arm_plan.seeds=[0,1,2]; verdict.json n=[3,3])Qwen3-0.6B/experiments/2026-06-18_qwen3-0.6b_imu1-deconfound-p1/verdict.jsonfound
Independent eval-harness (text-lm-v2) scoring of the three build checkpoints confirms the ordering on wikitext-2: modernized 27.80 < faithful 37.01 < partial-RoPE-0.25 38.08 PPL (and code_py 129.42 < 438.67 < 447.30).faithful: ppl.wikitext2_val.target 37.010055463333096, ppl.code_py.target 438.67295146042875 (noise floor wikitext floor_abs 1.3039039020038814); modernized (checkpoint_imu1_2tpp_step18000.pt): 27.79954726958199 / 129.422726138489 (wikitext floor_abs 1.0663419728457697); prope25: 38.079604272040854 / 447.30146774857917 (wikitext floor_abs 1.7040052364117315); all n_tokens 204600, suite_version text-lm-v2, dated 2026-06-16Qwen3-0.6B/experiments/2026-06-16_qwen3-0.6b_eval-modernized/eval/suite_results.jsonfound
RESULT.md's stated link between the two studies: the bundled IMU-1 result (−17.9% PPL at 1.19B tokens) could not attribute the gain, and the 42M-token single-variable run isolates the optimizer axis; the faithful 1.19B baseline's wikitext BPB anchor is 1.2256.'The project's IMU-1 matched-compute result (NorMuon bundled with ~5 other changes ... at 1.19B tokens, −17.9% PPL) had to mark optimizer attribution as an explicit limitation' (§3); 'faithful baseline Qwen3-0.6B/builds/2026-06-08_reproduce-faithful_qwen3-0.6b/ (wikitext BPB 1.2256 @ 1.19B tok)' (§6); '$the same NorMuon inside the full IMU-1 bundle at 1.19B gave only −17.9% PPL' (§4.3)Qwen3-0.6B/experiments/2026-06-16_qwen3_normuon-vs-adamw/RESULT.mdfound
Data composition · data-mix · 8 claims
dclm-edu vs FineWeb-Edu head-to-head: on code_py, dclm-edu improves BPB by +0.7034 (95% CI [+0.6639, +0.7430]), significant, n=3 seeds paired vs reused FineWeb-Edu controls.improvement_bpb = 0.7034377813053776; ci95 = [0.6639233630446848, 0.7429521995660704]; significant = true; baseline_mean = 2.638538404285876; treatment_mean = 1.9351006229804983; 3 seeds per armQwen3-0.6B/experiments/2026-06-24_qwen3-0.6b_data-dclm-vs-fineweb/verdict.jsonfound
dclm-edu vs FineWeb-Edu head-to-head: on wikitext2_val (English), dclm-edu is directionally worse by -0.0097 BPB (95% CI [-0.0255, +0.0061]), NOT significant.improvement_bpb = -0.0097280316070818; ci95 = [-0.025549693563240664, 0.006093630349077065]; significant = false; baseline_mean = 1.515714765474178; treatment_mean = 1.5254427970812598Qwen3-0.6B/experiments/2026-06-24_qwen3-0.6b_data-dclm-vs-fineweb/verdict.jsonfound
The 50/50 mix captures ~84% of the pure-dclm code win: mix code improvement 0.5904307914737172 / dclm code improvement 0.7034377813053776 = 0.8394 (83.94%).0.5904307914737172 / 0.7034377813053776 = 0.8393504118844001 -> 83.94% ~ '~84%'Qwen3-0.6B/experiments/2026-06-26_qwen3-0.6b_data-mix-composition/verdict.jsonfound
50/50 mix arm on code_py: BPB improvement +0.5904, 95% CI [+0.5532, +0.6277], significant, n=3.improvement_bpb = 0.5904307914737172; ci95 = [0.5531974784151051, 0.6276641045323293]; significant = true; treatment_bpb has 3 seed entries [2.0498830273156794, 2.050014838147228, 2.0444249729735695]; treatment_mean = 2.0481076128121587Qwen3-0.6B/experiments/2026-06-26_qwen3-0.6b_data-mix-composition/verdict.jsonfound
50/50 mix arm on wikitext2_val: BPB improvement +0.0161, 95% CI [-0.0005, +0.0327], NOT significant (English is preserved, no regression).improvement_bpb = 0.016087033990631383; ci95 = [-0.0005010071851609399, 0.032675075166423706]; significant = false; treatment_mean = 1.4996277314835467 vs baseline_mean = 1.515714765474178Qwen3-0.6B/experiments/2026-06-26_qwen3-0.6b_data-mix-composition/verdict.jsonfound
The first mix run was invalid due to a token-level shuffle that destroyed sequence structure; the fix makes the mix sequence-preserving (coherent halves concatenated, DataLoader shuffles at the packed-sequence level).Fix commit a576baa94afde60b7212c78778a0bb18d2968bbe ('Fix mix-arm data bug: token-level shuffle destroyed sequence structure (caught by implausible verdict)') removes the line 'np.random.default_rng(args.seed).shuffle(tr) # interleave the two sources' and adds the comment: 'concatenate the two COHERENT halves; the mix happens at the SEQUENCE level — PackedTextDataset packs seq_len windows (each from one source, coherent) and DataLoader(shuffle=True) randomizes their order. Do NOT shuffle tokens here: that destroys all sequence structure → token soup → garbage. (Verified bug, fixed.)'Qwen3-0.6B/experiments/2026-06-26_qwen3-0.6b_data-mix-composition/train_dataarm.pyfound
Iso-token/iso-FLOP budget: every arm (FineWeb control, dclm treatment, 50/50 mix) trains 2000 steps at 65,536 tokens/step (seq_len 4096 x micro_batch 4 x grad_accum 4) = 131,072,000 tokens per cell, 3 seeds per arm, only the data variable differs.run_arms.sh (both dirs): STEPS=2000; WARMUP=100; c5_evidence.json: budget.steps_per_cell = 2000, budget.tokens_per_cell = 131072000, budget.new_cells = 3; confound_check = {n_vars: 1, iso_flop: true, detail: 'identical model (596M) + identical token budget; only DATA differs -> train_flops identical, ratio 1.000'}; train_dataarm.py defaults: --seq_len 4096, --micro_batch 4, --grad_accum 4 (4096*4*4=65,536; 2000*65,536=131,072,000)Qwen3-0.6B/experiments/2026-06-24_qwen3-0.6b_data-dclm-vs-fineweb/c5_evidence.jsonfound
Both experiments' results are stamped with the versioned eval suite and use control-arm reuse per §C13 (FineWeb baseline = 2026-06-18 imu1-deconfound-p1 baseline checkpoints; mix experiment also reuses the dclm arm from 2026-06-24).suite_version = 'text-lm-v2' in verdict.json AND cohort_bpb.json of both experiments; control checkpoints are symlinks: checkpoint_control_seed{0,1,2}.pt -> ../2026-06-18_qwen3-0.6b_imu1-deconfound-p1/checkpoint_baseline_seed{0,1,2}.pt (dclm exp) and checkpoint_dclm_seed{0,1,2}.pt -> ../2026-06-24_qwen3-0.6b_data-dclm-vs-fineweb/checkpoint_treatment_seed{0,1,2}.pt (mix exp)Qwen3-0.6B/experiments/2026-06-26_qwen3-0.6b_data-mix-composition/verdict.jsonfound
Mid-training anneal & context gate · midtrain · 10 claims
The premium-mix anneal beats the iso-token control on code_py by +0.2716 BPB, 95% CI [+0.2641, +0.2792], significant, n=3 seeds per arm.improvement_bpb 0.27160749360360237, ci95 [0.2640563484201071, 0.2791586387870976], significant: true; treatment_mean 1.851990939811249 vs control_mean 2.1235984334148514; 3 per-seed values per armQwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/verdict.jsonfound
wikitext2 improves +0.0157 BPB, CI [+0.0140, +0.0174], significant — violating the pre-registered expected honest null on English.improvement_bpb 0.01573805476049084, ci95 [0.014040993622104418, 0.01743511589887726], significant: true; pre-registration in c5_evidence.json: "Expected: small code-BPB win only; null on English/downstream (pre-registered honest null, §C25)"Qwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/verdict.jsonfound
The iso-token control barely moved off the un-annealed base: +0.0050 code_py BPB and +0.0036 wikitext2 BPB — i.e. the LR-decay + 150M extra FineWeb tokens alone did ~nothing (the confound eaten).fineweb/code_py delta_vs_base 0.004996952805949473; fineweb/wikitext2_val delta_vs_base 0.0036343205041389215; both regressed: false; base BPB code_py 2.128595386220801, wikitext2 1.2256204566076285Qwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/verdict.jsonfound
Recipe: 1-sqrt cooldown from peak LR 2.5e-4 to end LR 2.5e-5 (10% floor), 2300 steps x 65,536 tok = 150.7M tokens/cell, ~13% of the 1.19B-token pretrain."1-sqrt cooldown peak_lr 2.5e-4 -> end_lr 2.5e-5 (10% floor), warmup 46, 2300 steps x 65,536 tok = 150.7M tok/cell (~13% of 1.19B pretrain), mb4 x ga4, AdamW(0.9,0.95) wd 0.01, bf16 seq 4096, chunked fp32 CE, compile on"Qwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/c5_evidence.jsonfound
ECL passkey ladder: the base's anomalous 4096-rung accuracy (0.025, i.e. ~0.03) is fixed to 0.40 by both arms; mix pooled 512-rung 0.8167 (~0.82) vs base 0.20; no arm regresses vs base at any rung; n=40/rung/cell with Wilson 95% CIs.base per_rung_accuracy[4096]=0.025 (Wilson95 [0.0044268315026814095, 0.1288136896347409]); mix pooled 4096 acc=0.4 and fineweb pooled 4096 acc=0.4; mix pooled 512 acc=0.8166666666666667 (Wilson95 [0.7379908648282967, 0.8756969964398578]) vs base 512=0.2; n_per_rung_per_cell=40; ladder_regressed_cells=[]Qwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/verdict.jsonfound
Step-0 context-extension gate FAILED: the base scores passkey 0.083 at its own 4096 trained window vs threshold 0.5, so context extension stays propose-only.accuracy_at_trained_len 0.08333333333333333, trained_len 4096, threshold 0.5, gate "FAIL", verdict "context-extension MOOT — base collapses before its trained window; propose-only"Qwen3-0.6B/experiments/2026-06-27_qwen3-0.6b_midtraining/step0_diagnostic.jsonfound
Robustness: excluding the twice-killed-and-resumed mix_seed0 cell entirely, the headline holds — code_py +0.2717, CI [+0.2333, +0.3100], significant."Robustness: excluding mix_seed0 entirely, the headline holds: code_py +0.2717, CI [+0.2333, +0.3100], significant (n=2 treatment, wide-CI warning)."research/ledger/runs/2026-06-30_qwen3-0.6b_midtrain-anneal.mdfound
The §C18 confound_check (n_vars=1, iso_flop=true) is recorded in the ledger.json runs[] entry for this run.confound_check: {"n_vars": 1, "iso_flop": true, "evidence": "single variable = DATA (mix vs fineweb), identical base ckpt + cooldown schedule + 2300 steps x 65,536 tok = 150.7M consumed tokens per cell (iso-FLOP exact by construction); §C5 evidence in c5_evidence.json"}research/ledger/ledger.jsonfound
(Adjacent) The constant-LR full continued-train sibling gave +0.5904 code_py BPB (CI [+0.5532, +0.6277]); the anneal retains ~46% of that effect.constant_lr_anchor.code_py_improvement_bpb 0.5904307914737172, code_py_ci95 [0.5531974784151051, 0.6276641045323293], regime "full continued-train at normal LR (NOT an anneal) — upper-bound reference only"Qwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/verdict.jsonfound
(Adjacent) §C25 gate: mid-training stage HARD-complete (all 3 required items present) → final_verdict win.c25_gate: verdict "win"; completeness: stage "mid-training", complete true, required=[effective_context_length_ruler, short_ctx_non_regression, anneal_gain_vs_iso_token_control], missing_hard=[], report_missing=["figure"], registry_version "v1", researched_on "2026-06-22"; why "HARD-complete and significant (§C25.3.5)"; final_verdict "win"Qwen3-0.6B/experiments/2026-06-30_qwen3-0.6b_midtrain-anneal/verdict.jsonfound
SFT & the eval-token confound · sft-confound · 7 claims
On the fixed held-out response-masked reasoning set, PPL improves from base 14.127 to 11.573 (SFT) and 11.582 (iso-FLOP control), i.e. −18.1% / −18.0% vs base.base reasoning_ppl_masked = 14.1267; SFT seeds = 11.5732 / 11.5722 / 11.5723 (mean 11.573); ctrl seeds = 11.5835 / 11.5819 / 11.5799 (mean 11.582); ledger table: SFT −2.554 (−18.1%), control −2.545 (−18.0%) vs base 14.127Qwen3-0.6B/experiments/2026-06-27_qwen3-0.6b_sft-3seed/reasoning_verdict.jsonfound
SFT−control separation: masked comparison +0.009 PPL CI95 [0.0045, 0.0139] significant, but the full-sequence cross-check gives −0.006 CI95 [−0.011, −0.0015] not significant — the two disagree, so the verdict is directional.sft_vs_control.improvement_ppl = 0.009199999999999875, ci95 = [0.004514755051701214, 0.013885244948298535], significant = true; sft_vs_control_fullseq.improvement_ppl = -0.006366666666670184, ci95 = [-0.011243400000003543, -0.0014899333333368266], significant = false; overall_verdict = "directional — masked and full-sequence comparisons disagree on significance; treat as not-yet-separable"Qwen3-0.6B/experiments/2026-06-27_qwen3-0.6b_sft-3seed/reasoning_verdict.jsonfound
The in-loop (advisory) verdict showed SFT beating the control by 0.68 PPL 'significant', but this was an eval-token confound (SFT scored on response tokens only, control on all tokens); re-scored on one fixed response-masked held-out set the gap collapses to ~0.01 PPL.verdict.json: sft_vs_control.improvement_ppl = 0.6786666666666665, ci95 [0.6760494287672351, 0.681283904566098], significant = true, with CONFOUND key: "in-loop control PPL is over ALL tokens, SFT PPL over RESPONSE tokens only — NOT directly comparable... Advisory, not the verdict."; ledger md: "the gap collapses 0.68 → ~0.01. The corrected verdict is the one of record."Qwen3-0.6B/experiments/2026-06-27_qwen3-0.6b_sft-3seed/verdict.jsonfound
Catastrophic-forgetting probe: held-out FineWeb-Edu PPL base 21.495 vs SFT 21.652 vs control 21.660, and wikitext2/code retention within noise (retained).reasoning_verdict.json forgetting: base_ppl = 21.4954, sft_mean = 21.6519, ctrl_mean = 21.6604; suite_results.json (sft_seed0) forgetting: wikitext2_val base 37.010055463333096 → target 37.08215923564533 (Δ +0.0721037723122322, floor_abs 1.3039039020038814, label "retained"), code_py 438.67295146042875 → 425.6893140677225 (Δ −12.983637392706271, floor_abs 280.56392586197546, label "retained"), summary "retained"Qwen3-0.6B/experiments/2026-06-27_qwen3-0.6b_sft-3seed/reasoning_verdict.jsonfound
Config: ~125M in-domain math (OpenR1-Math) tokens, 3 seeds per arm, peak_lr 5e-5 with cosine decay to 8e-8 (5% warmup), and an iso-FLOP --no_mask continued-pretrain control differing only in masking.c5_evidence.json recipe: "faithful base; response-masked CE via posttrain_losses.masked_sft_nll; peak_lr 5e-5 cosine->8e-8, warmup 5%, mb4 x ga32 (global 128), AdamW(0.9,0.95) wd 0.01, ~125M tok, compile on"; arm_plan: arms ["sft (response-masked)", "ctrl (--no_mask continued-pretrain)"], seeds [0,1,2], new_cells 6; single_variable: "masking only"Qwen3-0.6B/experiments/2026-06-27_qwen3-0.6b_sft-3seed/c5_evidence.jsonfound
Across-seed spread of the held-out masked reasoning PPL is ~0.001 within each arm (n=3).Ledger md line 38: "Seed spread within each arm ≈ 0.001 PPL (n=3)"; raw seeds — SFT: 11.5732/11.5722/11.5723 (max−min = 0.0010), ctrl: 11.5835/11.5819/11.5799 (max−min = 0.0036)research/ledger/runs/2026-06-27_qwen3-0.6b_sft-3seed.mdfound
Standard-suite general-domain evals: SFT ≈ control ≈ base on wikitext2 and code, both within the noise floor (comparable numbers from eval-harness, suite text-lm-v2).wikitext2_val: base 37.010055463333096, SFT seed0 37.08215923564533, ctrl seed0 37.084243181550235, floor_abs 1.3039039020038814, label "not significant"; code_py (codeparrot_clean_valid): base 438.67295146042875, SFT 425.6893140677225, ctrl 425.59445853983783, floor_abs 280.56392586197546, label "not significant"; suite_version "text-lm-v2", date "2026-06-30"Qwen3-0.6B/experiments/2026-06-27_qwen3-0.6b_sft-3seed/eval/sft_seed0/suite_results.jsonfound
RLVR pre-registered null · rlvr · 17 claims
Phase-1 go/no-go: the SFT checkpoint scores GSM8K pass@1 1.1% (Wilson CI [0.6%, 2.1%]) and pass@8 7%.acc 0.01125, ci_low 0.0059297687461997645, ci_high 0.021241582744848563; passk_chen2021 {"1": 0.01125, "8": 0.07}; solved_items 7 of n_items 100, n_samples 8, temp 0.8, max_new_tokens 256, band_seed 20260701Qwen3-0.6B/experiments/2026-07-01_qwen3-0.6b_rlvr-phase1-passk/phase1_passk.jsonfound
Phase-1: the SFT checkpoint scores MATH-500 L1-3 pass@1 1.5% / pass@8 10%.acc 0.015, ci [0.006892291919440893, 0.03233463358482672]; passk {"1": 0.015, "8": 0.1}; solved_items 5 of n_items 50Qwen3-0.6B/experiments/2026-07-01_qwen3-0.6b_rlvr-phase1-passk/phase1_passk.jsonfound
Phase-1: the pretrain base scores GSM8K 0.6% pass@1 / 5% pass@8.acc 0.00625, ci [0.0026724940266688486, 0.014546646226436234]; passk {"1": 0.00625, "8": 0.05}; solved_items 5/100Qwen3-0.6B/experiments/2026-07-01_qwen3-0.6b_rlvr-phase1-passk/phase1_passk.jsonfound
Phase-1: the pretrain base scores MATH-500 L1-3 2.25% pass@1 / 14% pass@8.acc 0.0225, ci [0.011881583530495034, 0.04220265755875813]; passk {"1": 0.0225, "8": 0.14}; solved_items 7/50Qwen3-0.6B/experiments/2026-07-01_qwen3-0.6b_rlvr-phase1-passk/phase1_passk.jsonfound
Phase-1 pre-registered GO rule was 'SFT solved_items >= 3 across the band'; 12 items solved -> decision GO."go_rule": "GO iff SFT solved_items >= 3 across the band (pre-registered)"; "decision": "GO"; "solved_total_sft": 12Qwen3-0.6B/experiments/2026-07-01_qwen3-0.6b_rlvr-phase1-passk/phase1_passk.jsonfound
Phase-2 GRPO trained for 300 steps (16 prompts x G=8 rollouts @ T=0.8, cap 256).health_grpo_seed0.jsonl has exactly 300 step records (step 1..300); train_grpo.py --steps default=300; c5_evidence recipe: '300 steps x 16 prompts x G=8 @ T=0.8 cap 256'Qwen3-0.6B/experiments/2026-07-02_qwen3-0.6b_grpo-phase2/health_grpo_seed0.jsonlfound
Phase-2 GRPO mean training reward stayed flat at ~0.9% verifier-correct over all 300 steps with no learning trend.overall mean of mean_reward = 0.009231 (min 0.0, max 0.0391); first-50 mean 0.00905, last-50 mean 0.007956; thirds 0.008737/0.009986/0.008970; OLS slope +5.8e-7/step (+0.0002 over 300 steps); kept_groups (DAPO) mean 13.49/16Qwen3-0.6B/experiments/2026-07-02_qwen3-0.6b_grpo-phase2/health_grpo_seed0.jsonlfound
Phase-2 GRPO per-token k3 KL vs the frozen SFT reference stayed at ~1.7e-4 magnitude throughout.kl field: mean 1.709e-4, min 0.0 (step 1, policy==ref), max 2.03e-4, final step 300 = 1.74e-4Qwen3-0.6B/experiments/2026-07-02_qwen3-0.6b_grpo-phase2/health_grpo_seed0.jsonlfound
The iso-generation-compute RFT control collected 0 verifier-correct completions (README wording).CONTRADICTED: train_rft_seed0.log final lines read '[14:52:11] RFT: 352 verifier-correct completions collected; masked-SFT pass' then '[14:52:31] RFT sft-pass 200/352' and '[14:52:51] DONE'. 352/38,400 rollouts = 0.917% — matching health_rft_seed0.jsonl overall mean_reward 0.009153 and the ~1% rollout-correct rate seen in supervision.Qwen3-0.6B/experiments/2026-07-02_qwen3-0.6b_grpo-phase2/train_rft_seed0.logcontradicted
The random-reward (Spurious-Rewards) gate arm trained at reward ~0.5, proving the update pipeline works.health_random_seed0.jsonl mean_reward: overall mean 0.501406, min 0.375, max 0.5938, final step 0.5234; kept_groups mean 15.88/16; loss mean +0.0281Qwen3-0.6B/experiments/2026-07-02_qwen3-0.6b_grpo-phase2/health_random_seed0.jsonlfound
Final paired eval, GSM8K: pass@1/pass@8 = SFT 1.12%/7%, GRPO 1.0%/7%, random 1.0%/6%, RFT 1.5%/11%; GRPO beats neither the SFT floor nor the random gate.verdict.json comparison[0]: sft_pass1 0.0112, grpo_pass1 0.01, random_pass1 0.01, rft_pass1 0.015; sft_pass8 0.07, grpo_pass8 0.07, random_pass8 0.06, rft_pass8 0.11; grpo_pass1_ci [0.0051, 0.0196], sft_pass1_ci [0.0059, 0.0212]; grpo_beats_sft_floor false, grpo_beats_random_gate falseQwen3-0.6B/experiments/2026-07-02_qwen3-0.6b_grpo-phase2/verdict.jsonfound
Final paired eval, MATH-500 L1-3: pass@1/pass@8 = SFT 1.5%/10%, GRPO 2.25%/16%, random 1.5%/10%, RFT 1.75%/12%; both gates fail (CIs overlap).verdict.json comparison[1]: sft_pass1 0.015, grpo_pass1 0.0225, random_pass1 0.015, rft_pass1 0.0175; sft_pass8 0.1, grpo_pass8 0.16, random_pass8 0.1, rft_pass8 0.12; grpo_pass1_ci [0.0119, 0.0422], sft_pass1_ci [0.0069, 0.0323]; grpo_beats_sft_floor false, grpo_beats_random_gate falseQwen3-0.6B/experiments/2026-07-02_qwen3-0.6b_grpo-phase2/verdict.jsonfound
Phase-2 verdict: directional (capped by design at n=1 seed per §C25), proceed_to_phase3 = false, pre-registered null confirmed."verdict": "directional"; "verdict_capped_reason": "n=1 seed; §C25 rlvr requires seed_ci (>=3 paired seeds) for a non-directional call"; "seeds": 1; "proceed_to_phase3": false; conclusion: "PREDICTED NULL CONFIRMED: GRPO does not beat the SFT floor; GRPO does not beat the random-reward gate. ... Do NOT spend the multi-seed cohort; the reasoning gain lives in SFT/distillation, not RL at this scale."Qwen3-0.6B/experiments/2026-07-02_qwen3-0.6b_grpo-phase2/verdict.jsonfound
The null was pre-registered before the run: plan.md §1 predicts RLVR at 0.6B is a methodology/negative-result artifact whose most likely true outcome is a null-to-tiny delta the control battery should refuse to call a win.§1 'HONEST verdict up front': 'At 0.6B on our ~10x-undertrained FineWeb-Edu→SFT base, RLVR is a methodology/negative-result artifact, not a reasoning win.' ... 'The most likely *true* outcome is a null-or-tiny attributable Dr.GRPO gain that the control battery should correctly **refuse to call a win**'. Header: 'researched_on 2026-07-01'.research/rlvr/plan.mdfound
Decontamination: both eval sets are 0-flagged against the 35,924 OpenR1 SFT problems (13-gram index)."decontam_target": "35924 OpenR1 SFT problems (of 35231 used uuids)"; gsm8k_test {n 1319, contaminated 0, clean 1319}; math500 {n 500, contaminated 0, clean 500}; ngram_n 13, flag_threshold 0.5, index_13grams 1138279; max_overlap_gsm8k 0.0, max_overlap_math500 0.0588research/datasets/math-eval-v1/decontam_report.jsonfound
Decontamination: 14 eval-overlapping prompts were dropped from the GRPO training prompt set.gsm8k_train {n 7473, eval_dropped 2, clean 7471, sft_flagged 0}; math_l13_train {n 3502, eval_dropped 12, clean 3490, sft_flagged 7} — total eval_dropped = 2 + 12 = 14research/datasets/grpo-math-prompts-v1/decontam_report.jsonfound
The answer extractor/verifier is pinned and version-stamped as math-acc-v1 in every result.research/eval_math_acc.py line 37: EXTRACTOR_VERSION = "math-acc-v1"; stamped as "extractor_version": "math-acc-v1" in every per-set block of phase1_passk.json and passk_{grpo,random,rft}.json, as "extractor": "math-acc-v1" at top level of all four + verdict.json, and in both ledger run metrics + both decontam reportsresearch/eval_math_acc.pyfound

Every number on this page fetches
from a pinned commit.

Hover any stat — the tooltip names the source file, the commit, and the script that generated it.

The flagship →Read the write-ups →