1.1190 BPB — 11L LeakyReLU² XSA4 PartialRoPE LNScale EMA ParallelMuon TTT#181

Open

manfromnowhere143 wants to merge 27 commits intoopenai:mainfrom

manfromnowhere143:main

Commits on Mar 18, 2026

Aweb Depth Recurrence: 4 blocks × 6 repeats = 24 effective layers at 768 dim
manfromnowhere143
committed
v2: Add SwiGLU activation + Quantization-Aware Training (QAT)
manfromnowhere143
committed
v4: Full stack — Depth Recurrence + SwiGLU + MoE + QAT + TTT
manfromnowhere143
committed
v5: Add Differential Attention (Microsoft ICLR 2025)
manfromnowhere143
committed
v6: BitNet 1.58-bit MOONSHOT — 50M params in 16MB
manfromnowhere143
committed
FINAL: 9 techniques stacked — LoRA + Multi-Token Prediction added
manfromnowhere143
committed

Commits on Mar 19, 2026

Commits on Mar 20, 2026

Commits on Mar 21, 2026

Commits on Mar 27, 2026