Refactor TitansMAG forward logic and fix memory state handling#32

Open

ryancinsight wants to merge 252 commits intomainfrom

titans-mag-forward-6565573961390046446

This pull request is big! We're only showing the most recent 250 commits

Commits on Sep 14, 2025

Merge branch 'main' of github.com:tekaratzas/RustGPT
tekaratzas
committed

Commits on Sep 16, 2025

Commits on Sep 18, 2025

Commits on Sep 19, 2025

Run cargo-fmt

Mrityunjai Singh
authored and
Mrityunjai Singh
committed
CI to check and run tests

Mrityunjai Singh
authored and
Mrityunjai Singh
committed

Commits on Sep 22, 2025

Commits on Sep 24, 2025

PR comments addressed
anshumanpatil
committed

Commits on Sep 25, 2025

Merge pull request #2 from anshumanpatil/feature/isolate-data-loading
tekaratzas
authored
New Feature: Count Total Parameters

Ari Kaufman
authored and
Ari Kaufman
committed

Commits on Sep 26, 2025

Adjustments to constant variables

Ari Kaufman
authored and
Ari Kaufman
committed
Merge branch 'main' into housekeeping
ben1009
authored
chore: fix lints
ben1009
committed
Calculate Parameter at Layer Level
Ari Kaufman
committed
Adjust import
Ari Kaufman
committed

Commits on Sep 29, 2025

Fix build

Ari Kaufman
authored and
Ari Kaufman
committed
Run a Cargo fmt

Ari Kaufman
authored and
Ari Kaufman
committed
Merge pull request #14 from arikaufman/feature/count-total-parameters
tekaratzas
authored

Commits on Sep 30, 2025

Commits on Oct 2, 2025

Refactor: Eliminate repeated vocabulary processing code by moving to vocab module
Thomas Roques
committed

Commits on Oct 3, 2025

fix formatting
Thomas Roques
committed
Merge pull request #17 from tomschelsen/dry
tekaratzas
authored

Commits on Oct 4, 2025

comment out the cov gha
ben1009
committed

Commits on Oct 5, 2025

Commits on Oct 9, 2025

chore: fix readme workflow badge
ben1009
committed

Commits on Oct 10, 2025

Merge pull request #19 from ben1009/readme
tekaratzas
authored

Commits on Oct 14, 2025

Commits on Oct 15, 2025

Commits on Oct 16, 2025

Add comprehensive tests for RMSNorm, RotaryEmbedding, Sliding Window Attention, and SwiGLU layers
ryancinsight
committed

Commits on Oct 17, 2025

Commits on Oct 18, 2025

Commits on Oct 19, 2025

Commits on Oct 20, 2025

Commits on Oct 21, 2025

Commits on Oct 23, 2025

Commits on Oct 25, 2025

Commits on Nov 5, 2025

Commits on Nov 8, 2025

feat(moh): implement SoftTopP head selection strategy for improved routing
ryancinsight
committed

Commits on Nov 9, 2025

Commits on Nov 10, 2025

Commits on Nov 15, 2025

Commits on Nov 17, 2025

Commits on Nov 18, 2025

Add training logs for diffusion model runs with detailed metrics
ryancinsight
committed

Commits on Nov 19, 2025

fix: correct indexing bug and add debug logging in PolyAttention; rename diffusion_speculative to speculative_config
ryancinsight
committed

Commits on Nov 20, 2025

Add training logs for diffusion model runs on 2025-11-20
ryancinsight
committed

Commits on Nov 22, 2025

Refactor layer handling and add zero_gradients method for improved gradient management
ryancinsight
committed

Commits on Nov 23, 2025

Refactor attention and model configuration to use RichardsGate
ryancinsight
committed

Commits on Nov 24, 2025

Commits on Nov 25, 2025

Implement deterministic random number generation module with global seed support
ryancinsight
committed

Commits on Nov 30, 2025

Refactor code to remove unused variables and improve clarity across multiple modules
ryancinsight
committed

Commits on Dec 5, 2025

feat: Optimize Richards module performance and memory usage
ryancinsight
committed

Commits on Dec 6, 2025

feat: Implement position-aware residual scaling based on Theorem 4 with comprehensive validation
ryancinsight
committed

Commits on Dec 7, 2025

feat: Implement advanced adaptive residuals for diffusion models
ryancinsight
committed

Commits on Dec 10, 2025

Implement unified adaptive residuals for transformer and diffusion models
ryancinsight
committed

Commits on Dec 12, 2025

Refactor Richards curve implementation for improved performance and memory efficiency
ryancinsight
committed

Commits on Dec 13, 2025

Commits on Dec 19, 2025

Add Real-Gated Linear Recurrent Unit (RG-LRU) and Multi-head RG-LRU layers
ryancinsight
committed

Commits on Dec 20, 2025

Add transformer layers and speculative sampling implementation
ryancinsight
committed

Commits on Dec 21, 2025

Commits on Jan 3, 2026

feat(transformer): add modular components for attention and feedforward processing
ryancinsight
committed

Commits on Jan 4, 2026

Refactor RichardsCurve and related components for f32 support
ryancinsight
committed

Commits on Jan 5, 2026

Add comprehensive documentation for Mamba and RG-LRU architectures; introduce modular transformer components and speculative sampling implementation
ryancinsight
committed

Commits on Jan 6, 2026

Commits on Jan 8, 2026

feat: Enhance ModelConfig with new residual training parameters
ryancinsight
committed

Commits on Jan 12, 2026

Refactor and enhance various components
ryancinsight
committed

Commits on Jan 13, 2026

feat(model): add Titan memory mechanism and checkpointing support
ryancinsight
committed

Commits on Jan 14, 2026

feat(ssm): add MoH variants for Mamba and Mamba2 layers
ryancinsight
committed

Commits on Jan 15, 2026

Commits on Jan 16, 2026

Commits on Jan 21, 2026

Commits on Jan 22, 2026

Commits on Jan 23, 2026

Commits on Jan 24, 2026

Commits on Jan 25, 2026

Commits on Jan 26, 2026

Commits on Jan 27, 2026