Non-Record: 11L Parallel Muon + LeakyReLU² MLP3x + Legal TTT (val_bpb 1.1253)#635

Closed

aryanbhosale wants to merge 15 commits intoopenai:mainfrom

aryanbhosale:submission/sota-11l-mlp35x

Commits on Mar 25, 2026

Non-record: 11L MLP3.5x LeakyReLU(0.5)^2 + Full SOTA Stack (mean val_bpb=1.1330, 8xH100)
arbyte77
committed
P0-P5: Full Hessian GPTQ, LZMA, pruning, TTT rewrite, lr/trigram tuning

arbyte77
and
claude
committed
Fix Hessian collection dtype mismatch (addmm_ requires matching dtypes)

arbyte77
and
claude
committed
Fix Hessian collection: use float32 on GPU instead of float64 on CPU

arbyte77
and
claude
committed
Fix: re-enable autocast for GPTQ calibration forward pass

arbyte77
and
claude
committed
Fix GPTQ: use numerically stable cholesky_inverse, handle dead columns

arbyte77
and
claude
committed
Revert to GPTQ-lite: full Hessian GPTQ conflicts with QAT

arbyte77
and
claude
committed
Disable magnitude pruning — investigating 0.155 BPB quantization degradation

arbyte77
and
claude
committed
Revert LZMA to zstd-22: LZMA gave larger artifacts (14.9MB vs ~10MB)

arbyte77
and
claude
committed
Reduce bigram 10240→2048, 3% pruning, remove dead GPTQ code

arbyte77
and
claude
committed
Disable magnitude pruning — incompatible with QAT-trained weights

arbyte77
and
claude
committed
Port Parallel Muon from merged SOTA: parameter banking + batched NS5

arbyte77
and
claude
committed
Use flash_attn_3 when available — expected ~2x speedup on H100

arbyte77
and
claude
committed
Match reference SOTA config: MLP 3.0x, XSA4, bigram 1536, GPU EMA

arbyte77
and
claude
committed
Record: 11L Parallel Muon + LeakyReLU² MLP3x + Legal TTT (val_bpb 1.1253, 3-seed)
arbyte77
committed