Non-Record: 11L Parallel Muon + LN Scale + LeakyReLU² MLP3x + Legal TTT — val_bpb 1.1215 (3-seed mean)

[aryanbhosale]

Non-Record: 11L Parallel Muon + LN Scale + LeakyReLU² MLP3x + Legal Score-First TTT

val_bpb = 1.1215 (3-seed mean, std 0.0002) | ~15.85 MB | 8×H100 SXM

3-Seed Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Seed	step_avg	steps	EMA bpb	Quantized bpb	TTT bpb
1337	88.8ms	6,759	1.1161	1.1238	1.1217
42	88.8ms	6,757	1.1158	1.1234	1.1213
2024	88.9ms	6,752	1.1160	1.1234	1.1215
Mean	88.8ms	6,756	1.1160	1.1235	1.1215

Key Techniques

1. Parallel Muon with parameter banking — 4 contiguous 3D banks, batched Newton-Schulz via torch.bmm, 3-phase async reduce-scatter/all-gather. ~89ms/step, ~6750 steps.
2. LN Scale — depth-dependent normalization: 1/sqrt(layer_idx+1). Stabilizes deep layers, -0.003 BPB.
3. LeakyReLU(0.5)² MLP 3x — preserves negative gradient flow.
4. Legal Score-First TTT (PR Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461/Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 recipe) — score each 32K-token chunk with sliding windows under inference_mode, then SGD(lr=0.002, momentum=0.9) for 3 epochs, all blocks unfrozen, cosine LR decay.
5. EMA(0.997) + SWA — EMA selected as best pre-quant weights (1.1160 BPB mean).
6. GPTQ-lite int6 + zstd-22 — per-row 5-percentile clip search, FP16 embedding passthrough.
7. Flash Attention 3, torch.compile(fullgraph=True), no DDP.

Architecture (26.8M params)

11L, 512d, 8H/4KV (GQA), MLP 3x, LN Scale, SmearGate, BigramHash(1536), Value Residual, Gated Attention, XSA4, Partial RoPE(16/64), U-Net skips, OrthoInit, tied embeddings, logit softcap 30.0.

Training

• Parallel Muon: lr=0.025, momentum 0.92->0.99/1500 steps, WD=0.04, Newton-Schulz 5 steps
• Adam for embeddings (lr=0.035) and scalars (lr=0.025)
• Batch 786,432 tokens, seq_len 2048, warmdown 3500 iters
• Late QAT via STE (final 15% wallclock)
• Gradient clipping 0.3

TTT Compliance

Legal score-first per PR #461/#549 framework:

• Every token scored BEFORE any weight update (enforced by torch.inference_mode)
• No training data access during evaluation
• No multi-epoch scoring -- each chunk scored exactly once
• Total eval time: ~530s (within 10 min limit)

Credits

• Parallel Muon / Parameter Banking: PR Record: Parallel Muon + Parameter Banking — 81.87ms/step, val_bpb 1.1247 (3-seed mean) #399 by @abaybektursun
• LeakyReLU²: PR Record: 11L EMA + Int6 + XSA + LeakyReLU² + Partial RoPE (val_bpb: 1.1309) #493 by @parinzee, PR Record: 11L XSA4 + LeakyReLU(0.5)² + Cosine TTT 50ep (val_bpb=1.0622) #518 by @sofiabod
• LN Scale: PR Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315/374 by @jfprincz
• TTT recipe: PR Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461 by @Christopher-Lee-McClendon (adapted: freeze=0)
• Base model stack: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 by @signalrush