Non-record: Empirical Bayes Adaptive TTT (val_bpb=1.1185)

[Robby955]

Summary

Verified val_bpb = 1.1185 (8xH100 SXM, 6909 steps at 86.8ms/step, artifact 15.81 MB).

Built on PR #589's frontier architecture (11L GEPA, VE128, XSA, SWA, Late Soft-Round QAT, Bigram, SmearGate). Our contribution is Empirical Bayes Adaptive Test-Time Training (EB-TTT): a per-layer gradient SNR scaling method that improves TTT by 0.0015 BPB over baseline SGD.

Results

Metric	PR #589 Baseline (our repro)	+ EB-TTT (this PR)	Delta
Training steps	6801	6909	+108
Post-EMA BPB	1.1375	1.1356	-0.0019
Post-quant BPB	1.1458	1.1443	-0.0015
Post-TTT BPB	1.1200	1.1185	-0.0015
Artifact size	15.83 MB	15.81 MB	-0.02 MB

Research Contribution: EB-Adaptive TTT

Standard TTT applies uniform SGD learning rate to all layers. We observe that different layers have different gradient signal-to-noise ratios (SNR) when adapting to local validation chunks. EB-TTT scales each layer's gradient by its SNR:

lambda_i = clip(|E[grad_i]| / std(grad_i), 0.3, 3.0)

Interpretation: High SNR (consistent gradient direction) means the layer has a clear adaptation signal — amplify it. Low SNR (noisy gradients) means the layer should stay closer to the trained prior — attenuate it.

Learned Layer Scales (stable across runs)

blocks.0=0.54  blocks.1=0.61  blocks.2=0.61  blocks.3=0.61  blocks.4=0.63
blocks.5=0.65  blocks.6=0.66  blocks.7=0.68  blocks.8=0.65  blocks.9=0.58  blocks.10=0.51
ve_layer_scales: 3.00 (saturated at cap)

Middle layers (blocks.5-8) adapt most aggressively, boundary layers (blocks.0, 10) stay closer to prior. This matches intuition: early/late layers handle general tokenization/prediction while middle layers specialize to local data distribution.

Additional TTT Improvements

• Embedding freeze during TTT (TTT_FREEZE_EMBEDDINGS=1): Prevents vocabulary embedding distortion when adapting to local 32K-token chunks. Token embeddings, bigram hash, and lm_head are frozen.
• TTT burst with EMA: 2-epoch burst replay at 0.1x LR before sliding-window TTT.

Ablation Results (all on 8xH100 SXM)

Run	Config	Final BPB	Notes
1	PR #589 base + VRL	1.1200	Previous best
2	+ MTP + James-Stein shrinkage	1.1312	MTP + J-S hurt
7	EB-TTT (H100 NVL, 3540 steps)	1.1526	TTT delta -0.040 (54% better than SGD)
8	EB-TTT, no burst (SXM)	1.1192	Best absolute, artifact over 16MB
9	EB-TTT + burst (SXM)	1.1185	Best valid submission

Negative Results

• VRL (Value Residual Learning) on GEPA: +0.002 worse, collides with VE128
• MTP (Multi-Token Prediction): Dilutes gradient in compute-limited regime
• James-Stein TTT shrinkage: Over-attenuates (TTT is underfitting, not overfitting)
• AdamW for TTT: Diverges catastrophically
• EVAL_SEQ_LEN=4096: RoPE NTK scaling doesn't generalize (1.53 BPB)
• Output-aware quantization: Same BPB, bigger artifact

Architecture

11 layers, 512-dim, 8 heads, 4 KV heads. GEPA attention, VE128, XSA (last 4 layers), SWA, Late Soft-Round QAT, BigramHash, SmearGate. int6+zstd quantization. Single-pass score-first TTT with EB-adaptive per-layer scaling.

Credits

@RoyiRa PR #589 (base architecture), @thwu1 PR #180, @unnir PR #162, @JoeProAI PR #462 (sequential TTT, SwiGLU), ResFormer (VRL concept), @ndokutovich PR #486 (per-layer LR). EB-TTT concept inspired by Empirical Bayes shrinkage estimation applied to gradient SNR.