Record*: val_bpb=0.978 BPB — Goldfish ML Autonomous Research (100ep Cosine *leaky* TTT)

[lukacf]

Edit: Record* = Yes, but uses suspect leaky TTT, so not a clean solution.

Summary

• val_bpb = 0.9789 (3-seed mean, sliding window stride=64, std=0.0015)
• 15.51 MB artifact, 8xH100 SXM
• Built on PR Non-record: 11L EMA + TTT(20ep,freeze=0) + 15-run ablation study — val_bpb=1.1213 (3-seed) #398/Record: 11L EMA + AdamW TTT 10ep (mean val_bpb=1.1027) #442 baseline
• Training: 600s. TTT: 1463s (100 epochs, CosineAnnealingLR)

Technique

CosineAnnealingLR applied to AdamW TTT optimizer, decaying lr from 0.001 to 0.00001 over 100 epochs. Prevents position-specific overfitting that limits constant-lr TTT to ~30 epochs. Three lines of code.

Methodology

Experiments were run autonomously by an AI coding agent using Goldfish ML for experiment orchestration and provenance tracking. The agent executed the full research loop — hypothesis, implementation, launch, monitoring, analysis, iteration — without human intervention on the training code.

Seven experiments were completed in ~2 hours wall-clock time, progressing from baseline replication (1.085 BPB) through the cosine LR discovery (1.018) to the final result (0.978). Dead ends (weight decay, BigramHash scaling, Value Residual) were documented automatically. Full experiment lineage and compressed timeline in the submission README.

Files

• train_gpt.py — standalone training script
• train.log — full log (seed 1337)
• submission.json — 3-seed results
• README.md — detailed write-up with experiment timeline and provenance

[lukacf]

10 min training time, but I missed the 10 min eval time limit, which I now see this violates by a factor of 2. So goes into the "non leaderboard" bucket.

[lukacf]

I should add that the way TTT used is clearly suspect wrt. eval tokens leaking into training. Educational, but not a clean solution.

[lukacf]

Closing PR (TTT not valid)