Record: N-gram Backoff + VRL + LeakyReLU² — val_bpb 0.9642 (3-seed mean)

[anthony-maio]

Summary

val_bpb = 0.9642 (3-seed mean, std 0.0002) | ~15.95 MB | 8×H100 SXM

3-Seed Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Seed	step_avg	steps	Pre-ngram bpb	Post-ngram bpb	ng_helped	Artifact
1337	88.7ms	6,765	1.1225	0.9640	38.5%	15,981,848
42	88.6ms	6,772	1.1224	0.9641	38.6%	15,904,632
2025	88.6ms	6,776	1.1231	0.9644	38.6%	15,974,308
Mean	88.6ms	6,771	1.1227	0.9642 (std 0.0002)	38.6%

All artifacts under 16,000,000 bytes. All 3 train logs attached.

Key Innovation: Multi-Order N-gram Backoff Cache

Backward-looking n-gram cache built causally from already-scored tokens. Zero artifact cost.

Entropy-Adaptive Alpha: alpha = 0.05 + 0.55 * sigmoid(2*(H-4)). Low alpha when neural model is confident, high alpha when uncertain.

Multi-Order Backoff (2-7gram): Highest matching order wins. 4M hash buckets per order. min_count=2 gate. Raw count ratios, no smoothing.

Compliance: Score-first — every token scored under torch.inference_mode() before any table update. N-gram tables built from already-scored tokens only. No training data access during eval. No oracle selection.

Training Architecture

PR #414 base + LeakyReLU² + VRL + lzma:
11L, 512d, 8H/4KV GQA, LeakyReLU(0.5)² MLP 3×, VRL, VE128, BigramHash(2048), XSA4, Partial RoPE 16/64, LN Scale, SmearGate, U-Net skips, EMA(0.997) + Tight SWA, Late QAT, GPTQ-lite int6 + lzma, FA3 Hopper, Muon WD=0.04

Credits

• N-gram backoff: PR Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727 by @Asukabot0
• Base model: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 by @signalrush
• LeakyReLU²: PR Record: 11L EMA + Int6 + XSA + LeakyReLU² + Partial RoPE (val_bpb: 1.1309) #493 by @parinzee, PR Record: 11L XSA4 + LeakyReLU(0.5)² + Cosine TTT 50ep (val_bpb=1.0622) #518 by @sofiabod
• VRL: ResFormer (arXiv:2410.17897), PR Record: 11L VRL + LeakyReLU² + Full GPTQ (3-seed mean val_bpb=1.1175) #569 by @gowtham0992

Test plan

• Seed 1337: 0.9640 bpb, 15.98MB valid
• Seed 42: 0.9641 bpb, 15.90MB valid
• Seed 2025: 0.9644 bpb, 15.97MB valid
• 3-seed mean: 0.9642, std 0.0002
• All train logs attached
• All artifacts under 16,000,000 bytes
• Score-first compliance verified

[Copilot]

Pull request overview

This PR adds a set of “triton-kernels” skill documents (duplicated across multiple agent/tooling directories) plus an MCP server configuration, but it does not include any of the model/architecture changes described in the PR title/description (N-gram backoff cache, VRL, LeakyReLU², etc.).

Changes:

• Add Triton kernel optimization “skill” documentation (core SKILL + specialized guides for fused norms/epilogues, memory efficiency, and quantized block-scaled GEMM) across multiple agent directories.
• Add MCP server configuration for colab-proxy-mcp.

Reviewed changes

Copilot reviewed 148 out of 338 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
.mcpjam/skills/triton-kernels/SKILL.md	Adds Triton kernel “skill” overview and references to specialized guides.
.mcp.json	Adds MCP server config to launch colab-proxy-mcp via uvx from a Git URL.
.kode/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.kode/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Adds quantized block-scaled GEMM guide.
.kode/skills/triton-kernels/triton-memory-efficient-patterns.md	Adds memory-efficient Triton patterns guide.
.kode/skills/triton-kernels/triton-fused-normalizations.md	Adds fused normalization kernels guide.
.kode/skills/triton-kernels/triton-fused-epilogue-kernels.md	Adds fused epilogue (attention/GEMM) patterns guide.
.kiro/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.kiro/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.kiro/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.kiro/skills/triton-kernels/triton-fused-normalizations.md	Duplicates fused normalization kernels guide.
.kiro/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.kilocode/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.kilocode/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.kilocode/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.kilocode/skills/triton-kernels/triton-fused-normalizations.md	Duplicates fused normalization kernels guide.
.kilocode/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.junie/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.junie/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.junie/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.junie/skills/triton-kernels/triton-fused-normalizations.md	Duplicates fused normalization kernels guide.
.junie/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.goose/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.goose/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.goose/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.goose/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.factory/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.factory/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.factory/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.factory/skills/triton-kernels/triton-fused-normalizations.md	Duplicates fused normalization kernels guide.
.factory/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.crush/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.crush/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.crush/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.crush/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.continue/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.continue/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.continue/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.continue/skills/triton-kernels/triton-fused-normalizations.md	Duplicates fused normalization kernels guide.
.continue/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.commandcode/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.commandcode/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.commandcode/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.commandcode/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.codebuddy/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.codebuddy/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.codebuddy/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.codebuddy/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.claude/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.claude/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.claude/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.claude/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.agents/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.agents/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.agents/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.agents/skills/triton-kernels/triton-fused-normalizations.md	Duplicates fused normalization kernels guide.
.agents/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.
.agent/skills/triton-kernels/SKILL.md	Duplicates the Triton kernel “skill” overview.
.agent/skills/triton-kernels/triton-quantized-block-scaled-gemm.md	Duplicates quantized block-scaled GEMM guide.
.agent/skills/triton-kernels/triton-memory-efficient-patterns.md	Duplicates memory-efficient Triton patterns guide.
.agent/skills/triton-kernels/triton-fused-normalizations.md	Duplicates fused normalization kernels guide.
.agent/skills/triton-kernels/triton-fused-epilogue-kernels.md	Duplicates fused epilogue patterns guide.

Comments suppressed due to low confidence (1)

.kode/skills/triton-kernels/SKILL.md:1

• This PR introduces the same 'triton-kernels' skill content replicated across many directories (.kode/.kiro/.kilocode/.junie/.goose/.factory/.crush/.continue/.commandcode/.codebuddy/.claude/.agent/.agents/.mcpjam). Keeping many identical copies will be brittle (fixes like typos/API updates must be applied everywhere). If these are meant to stay in sync, consider centralizing the source (single canonical doc + symlinks or a generation step) so updates are made once.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The MCP server is installed/executed directly from a moving Git URL, which is a supply-chain risk and can make builds non-reproducible. Pin the dependency to a specific tag/commit SHA (e.g., git+...@<sha>) and consider documenting/validating the allowed source to avoid unexpected remote code changes.

[anthony-maio]

Cleaning up — too many files leaked into diff.