Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed)#727
Closed
Asukabot0 wants to merge 1 commit intoopenai:mainfrom
Closed
Conversation
… 3-seed) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Congrats Asukbot0, nice work. I believe I was the first to contribute the N-gram eval cache technique PR #659 to the contest |
newjordan
pushed a commit
to newjordan/parameter-golf-1
that referenced
this pull request
Mar 25, 2026
Multi-order backoff (2-7) + entropy-adaptive alpha on 11L/512d U-Net. All 3 seeds sub-1.0. GPTQ calibration inside training phase. Seeds: 42=0.9631, 2045=0.9620, 7=0.9624, mean=0.9625 Credits: @deanbrr openai#659, @Asukabot0 openai#727, @signalrush openai#414 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Asukabot0
added a commit
to Asukabot0/parameter-golf
that referenced
this pull request
Mar 25, 2026
10 defaults were wrong (inherited from old PR#398 base): - MATRIX_LR: 0.04 -> 0.025 - SCALAR_LR: 0.04 -> 0.025 - TIED_EMBED_LR: 0.05 -> 0.035 - SWA_ENABLED: 1 -> 0 - XSA_LAST_N: 0 -> 11 - LEAKY_RELU: 0 -> 1 - MUON_MOMENTUM: 0.95 -> 0.99 - MUON_MOMENTUM_WARMUP_START: 0.85 -> 0.92 - MUON_MOMENTUM_WARMUP_STEPS: 500 -> 1500 Previous PR openai#727 runs worked because env vars were passed manually. After cloud restart, defaults kicked in producing wrong model. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4 tasks
8 tasks
anthony-maio
added a commit
to anthony-maio/parameter-golf
that referenced
this pull request
Mar 25, 2026
Backward-looking n-gram cache built from already-scored tokens. Multi-order backoff (2-7gram) with entropy-adaptive alpha mixing. Zero artifact cost. Expected -0.07 to -0.16 bpb gain. NGRAM_ENABLED=1 to activate. NGRAM_ORDER=7 (default). Disabled by default — legality still being debated (issue openai#677). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6 tasks
5 tasks
newjordan
pushed a commit
to newjordan/parameter-golf-1
that referenced
this pull request
Mar 26, 2026
Multi-order backoff (2-7) + entropy-adaptive alpha on 11L/512d U-Net. Two seeds sub-1.0. GPTQ calibration inside training phase. 3-seed: 1337=1.0217, 42=0.9631, 2045=0.9620, mean=0.9823 Credits: @deanbrr openai#659, @Asukabot0 openai#727, @signalrush openai#414 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4 tasks
6 tasks
4 tasks
6 tasks
6 tasks
anthony-maio
added a commit
to anthony-maio/parameter-golf
that referenced
this pull request
Mar 26, 2026
Key changes from failed Hedge implementation: - Entropy-adaptive alpha: 0.05 + 0.55 * sigmoid(2*(H-4)) (low when neural confident, high when uncertain) - Backoff: highest matching n-gram order wins (not mixture) - 4M hash buckets per order (was 65K) — matches PR openai#727 - Orders 2-7 (was unigram/bigram/trigram only) - Score-first: update cache AFTER scoring each segment - min_count=2 gate: n-gram only active with sufficient data - No smoothing: raw count ratios (not add-k) - Per-GPU contiguous windows with persistent cache Hedge failed because it learned a GLOBAL weight — neural always wins on average. PR openai#727's per-token entropy-adaptive alpha works because it only trusts n-grams when the neural model is uncertain. Progress logs show ng_helped% — fraction of tokens where n-gram mixing reduced NLL vs neural-only. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
7 tasks
6 tasks
This was referenced Mar 26, 2026
Fraser-Greenlee
added a commit
to Fraser-Greenlee/parameter-golf
that referenced
this pull request
Mar 26, 2026
Key insight: n-gram cache is worth 50-80x more than any architecture change (PR openai#727: 1.12 → 0.97 BPB). Our E16/E23 cache failures had specific fixable bugs (add-k smoothing, λ too high, per-token Python loops). New experiments: - E35: N-gram cache rewrite with Stupid Backoff (HIGHEST PRIORITY) - E36: XSA on all 11 layers (flag flip, adopted by frontier submissions) - E37: Full GPTQ + selective pruning Also documents which experiments are conclusively dead at this scale. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
anthony-maio
added a commit
to anthony-maio/parameter-golf
that referenced
this pull request
Mar 26, 2026
Key changes from failed Hedge implementation: - Entropy-adaptive alpha: 0.05 + 0.55 * sigmoid(2*(H-4)) (low when neural confident, high when uncertain) - Backoff: highest matching n-gram order wins (not mixture) - 4M hash buckets per order (was 65K) — matches PR openai#727 - Orders 2-7 (was unigram/bigram/trigram only) - Score-first: update cache AFTER scoring each segment - min_count=2 gate: n-gram only active with sufficient data - No smoothing: raw count ratios (not add-k) - Per-GPU contiguous windows with persistent cache Hedge failed because it learned a GLOBAL weight — neural always wins on average. PR openai#727's per-token entropy-adaptive alpha works because it only trusts n-grams when the neural model is uncertain. Progress logs show ng_helped% — fraction of tokens where n-gram mixing reduced NLL vs neural-only. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
abaybektursun
added a commit
to abaybektursun/parameter-golf
that referenced
this pull request
Mar 26, 2026
- Base model is ValCalib GPTQ (1.1142 BPB), not PR openai#549 (1.1194) - Remove stale "not yet deployed" / "we estimate" for EXP-11 - Note α=0.80 (939s) exceeds 600s budget - Fix PR openai#727 score to 0.9674, PR openai#788 to 0.9059 - Fix PR openai#596 BPB to 0.6430 - "Approved" → "Technique deemed legal" for closed PRs - Add bucket sweep and per-token overhead proposal - Replace "neural" with "base LM" throughout Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sunnypatneedi
pushed a commit
to sunnypatneedi/parameter-golf
that referenced
this pull request
Mar 27, 2026
- Update merged SOTA to 1.1194 (abaybektursun, was 1.1228 signalrush) - Add competition strategy pivot: n-gram eval cache now dominates (~0.02-0.97 bpb) - Document PR openai#727 (0.9674), openai#741 (0.9850), openai#945 (0.0274), openai#961 (0.0881) findings - Add Lessons Learned entries 17-20 on n-gram dominance + memorization risk - Update Technique Reference table with n-gram entries https://claude.ai/code/session_01Bpr2fKEnkNQmNKno8EnxWF
9 tasks
Contributor
|
Thanks for your submission! Unfortunately, it's disallowed due to the use of hashed n-gram caches, which do not renormalize correctly / correctly reweight the LM's token distribution, look ahead to the target token to mix probabilities and therefore leak eval tokens. Please refer to the long discussion about this under the issues tab for more details, and please submit more runs in the future! |
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Results (3-seed validation)
Technique
Architecture: 11L, 512d, GQA 8H/4KV, MLP 3x, LeakyReLU(0.5)², XSA-all(11), Value Residual, Gated Attention, SmearGate, BigramHash(4096), Partial RoPE(16/64), LN Scale, EMA(0.997). Tied embeddings. Muon optimizer.
N-gram eval cache — two key improvements over prior work:
Multi-order backoff (orders 2–7): Instead of a single fixed order, we attempt the highest order first and cascade down on miss. This dramatically improves coverage vs a fixed 7-gram.
Entropy-adaptive alpha:
alpha = 0.05 + 0.55 * sigmoid(2 * (H - 4.0)). When the model is uncertain (high entropy), we trust n-gram statistics more; when confident (low entropy), we trust the LM. This replaces the fixed alpha=0.40 used in prior approaches.Compliance
Ablation
Comparison with prior submissions
Training
Built on modded-nanogpt. Credits: PR #315, #609, #493, #518, #413, #674, #702.