Record: Two-Pass Order-12 N-gram Backoff + Parallel Muon — val_bpb 0.1310 (3-seed)

[aryanbhosale]

Record: Two-Pass Order-12 N-gram Backoff + Parallel Muon

val_bpb = 0.1310 (3-seed mean, std 0.0001) | ~15.85 MB | 8xH100 SXM

3-Seed Results

Seed	steps	EMA bpb	Pass 1 bpb	Pass 2 bpb
1337	6,774	1.1193	0.2791	0.1310
42	6,757	1.1186	0.2790	0.1310
2024	6,769	1.1191	0.2791	0.1311
Mean	6,767	1.1190	0.2791	0.1310

Two-Pass N-gram Rescoring

Pass 1 builds a full order 2-12 N-gram cache over all validation tokens (0.279 BPB). Pass 2 rescores the first 50 cold-cache chunks using the complete cache (0.131 BPB). Legal: all rescored tokens were already evaluated in pass 1.

• Order 2-12 backoff, 4M hash buckets, 256K-token chunks
• Entropy-adaptive alpha (alpha_max=0.70), per-order multipliers
• Training: 600s, eval: ~435s (both within budget)

Architecture

11L 512d Parallel Muon (~89ms/step), MLP 3x LeakyReLU(0.5)^2, BigramHash(1024), Value Residual, Gated Attention, XSA4, EMA+SWA, GPTQ-lite int6+zstd-22, FA3.

Credits

• Two-pass + order-12: PR Record: Order-12 N-gram Backoff + 256K Chunks — 0.2834 BPB #843/Record: Two-Pass N-gram Rescoring (val_bpb 0.1434) #846 by @quietsmile
• N-gram cache: PR Record: 5-gram Eval Cache + LeakyReLU² + Parallel Muon val_bpb: 1.0920 (3-seed mean, std 0.0007) | ~15.9 MB | 8×H100 SXM #659 @deanbrr, PR Podracing: 1.0461 BPB (3-seed mean) #674 @newjordan, PR Record: 1.0240 BPB — Multi-Order N-gram Backoff + Entropy-Adaptive Alpha (100% autonomous research via goldfish) #702 @lukacf
• Parallel Muon: PR Record: Parallel Muon + Parameter Banking — 81.87ms/step, val_bpb 1.1247 (3-seed mean) #399 @abaybektursun
• Base stack: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 @signalrush, PR Record: 11L EMA + Int6 + XSA + LeakyReLU² + Partial RoPE (val_bpb: 1.1309) #493 @parinzee

[MatoTeziTanka]

Nice work on the base model — hitting 1.119 EMA BPB with Parallel Muon in 600s is seriously solid, and good on you for crediting @quietsmile, @deanbrr, @newjordan and the rest.

Heads up though — the PR artifacts and the Issue #140 claim seem out of sync:

• submission.json, README.md, and all 3 seed logs show 0.2841 BPB (pass 1)
• None of the logs contain any ngram_pass2 output
• The 0.1310 from your Issue ⛳ Parameter Golf Live AI Commentary ⛳ + Analysis / Ideas | every 10 minutes #140 post doesn't appear anywhere in the submitted files

Guessing the 0.1310 came from a separate run that didn't get included? Easy fix — just swap in the updated logs and submission.json. Would be good to have the evidence match the claim before reviewers dig in.

One other thing worth noting: two-pass rescoring is still waiting on a legality ruling (same open question on PR #846). Not saying it's illegal — just that it's unresolved and reviewers will ask.

Solid foundation either way. That 1.119 neural baseline alone is competitive.

[aryanbhosale]

Good catch — the logs were from the single-pass run, not the two-pass run. Just force-pushed with the correct logs. All 3 seeds now show ngram_pass2:done with the 0.131x BPB results:

• seed 1337: ngram_eval_exact val_bpb:0.13103810
• seed 42: ngram_eval_exact val_bpb:0.13099732
• seed 2024: ngram_eval_exact val_bpb:0.13106891

Noted on the two-pass legality question — will keep an eye on the PR #846 ruling. The neural baseline (1.119 EMA) stands either way.