Record: 11L Full GPTQ + Multi-Order N-gram Backoff (fixed-alpha 0.9757 / entropy-adaptive 0.9605, 3-seed)

[raahilshah]

Summary

Fixed alpha (safest): 3-seed mean val_bpb = 0.9757 (std=0.0002)
Entropy-adaptive alpha: 3-seed mean val_bpb = 0.9605 (std=0.0003)

15.92 MB | 8xH100 SXM | Training 596s/600s

3-Seed Results

Seed	Neural-only	Fixed alpha (a=0.40)	Entropy-adaptive	Artifact	GPTQ Budget
1337	1.11719	0.97558	0.96027	15,921,027 B	596s/600s
42	1.11715	0.97562	0.96029	15,929,323 B	596s/600s
7	1.11787	0.97602	0.96082	15,922,059 B	596s/600s
Mean	1.11740	0.97574	0.96046
Std	0.00041	0.00024	0.00031

Two Variants

Variant 1: Fixed alpha (safest legal)

• NGRAM_ENTROPY=0, constant alpha=0.40
• Blend: p = 0.60 * model + 0.40 * ngram — same weight for every token
• 3-seed mean: 0.9757

Variant 2: Entropy-adaptive alpha

• NGRAM_ENTROPY=1, alpha = 0.05 + 0.55 * sigmoid(2 * (H - 4.0))
• Alpha depends on model output entropy H only (never on the true token)
• 3-seed mean: 0.9605

Key Techniques

Technique	Description
Full Hessian GPTQ	64-batch calibration within 600s training budget (reserved 14s)
Multi-order n-gram backoff	Orders 2-7, highest available order used first
Fixed/entropy-adaptive blend	Model + n-gram probability interpolation
Backward-looking cache	Counts updated AFTER scoring each window

Compliance

• Training: 586s training + 10s GPTQ = 596s (within 600s). No training data accessed during eval.
• GPTQ calibration: Uses training data within reserved training budget, NOT during eval
• Eval: ~86s sliding + ~130s cached = ~216s (within 600s)
• N-gram cache: Backward-looking only, no oracle selection, no true-token peeking
• Fixed alpha: No data-dependent weighting whatsoever
• Entropy-adaptive: Depends on model output distribution only, not ground truth
• Artifacts: All seeds under 16,000,000 bytes

Architecture

11L, 512d, GQA 8H/4KV, LeakyReLU(0.5)^2 MLP 3x, XSA-all, VE128, BigramHash(2048), Partial RoPE 16/64, LN Scale, SmearGate, U-Net skips, EMA(0.997), Parallel Muon, Full Hessian GPTQ int6 + LZMA.

Ablation (seed 1337)

Configuration	val_bpb	Delta
Neural-only (no cache)	1.1172	baseline
Fixed 7-gram only, alpha=0.40	1.0258	-0.0914
Multi-order backoff (2-7), fixed alpha	0.9756	-0.1416
Multi-order backoff (2-7), entropy-adaptive	0.9603	-0.1569

Credits

• Base: PR Non-record: 11L XSA-all + Full GPTQ + Selective Pruning (val_bpb=1.1154, 3-seed) #609 @saml212, PR Record: Full GPTQ + LeakyReLU² + Parallel Muon + BigramHash 3072 (val_bpb 1.1163, 3-seed mean) #593 @abaybektursun
• N-gram cache: PR Record: XSA-all + LeakyReLU² + VR + GA + 7-gram cache (val_bpb=1.0337) #715, PR Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727