Esp32s3 onboard neural net support

[Nickjgniklu]

Esp32s3 has decent support for vector operations check out for examples.

Adding a new config where an esp32 is the webserver and runs the neural net onboard would be optimal.

Do you think we can make a stand alone esp32s3 version?

@ruvnet

[ruvnet]

Great suggestion! We've been building exactly in this direction. Here's what's coming on the feat/complete-vendor-and-edge-intel branch (targeting PR #110):

On-Device Intelligence — Already Implemented

We now have 31 WASM modules that run directly on the ESP32-S3 via a WASM3 runtime. No server needed — the ESP32 reads WiFi signals and makes decisions locally in under 10 ms per frame.

What's working today

• WASM3 runtime on ESP32-S3 — hot-loadable modules, no reflash needed. Upload a new .wasm file over-the-air and it starts processing immediately.
• 24 vendor-integrated signal processing modules — attention, coherence gating, sparse signal recovery, multi-person tracking, gesture learning, anomaly detection, self-healing mesh, and more. All no_std Rust compiled to wasm32-unknown-unknown.
• 7 core modules — gesture recognition, coherence filtering, adversarial detection, intrusion detection, occupancy counting, vital sign trending, and RVF container parsing.
• 243 unit tests passing, security audited (15 findings addressed).

Key modules relevant to your request

Module	What It Does
Flash Attention (sig_flash_attention.rs)	Tiled attention over subcarrier groups — lightweight version of transformer attention
EWC Lifelong Learning (lrn_ewc_lifelong.rs)	Learns new tasks without forgetting old ones — Elastic Weight Consolidation on-device
Spiking Neural Tracker (spt_spiking_tracker.rs)	32 LIF neurons with STDP learning for zone tracking — biologically-inspired, runs in <5ms
Micro HNSW (spt_micro_hnsw.rs)	64-vector nearest-neighbor graph for classifying CSI snapshots against known room states
DTW Gesture Learn (lrn_dtw_gesture_learn.rs)	Users teach custom gestures by repeating them 3 times — 16 template slots, DTW matching
GOAP Autonomy (tmp_goap_autonomy.rs)	The ESP32 decides which modules to activate based on conditions — no server commands needed

Architecture

ESP32-S3 (925 KB firmware)
├── WASM3 runtime (Tier 3 — ADR-039)
├── Host API: 12 functions (phase, amplitude, vitals, events)
├── Hot-loadable .wasm modules (5-30 KB each)
├── Tier 2 vitals: breathing, heart rate, presence, fall detection
└── UDP streaming to aggregator + local decisions

The neural net doesn't run as a traditional TensorFlow model — instead, we use specialized signal processing modules (attention, spiking neurons, graph algorithms) that fit in the ESP32's memory and run within the frame budget. The WASM sandbox means modules can't crash the firmware, and you can swap them out without reflashing.

Standalone ESP32 mode

Yes — with the GOAP autonomy module, the ESP32 can operate fully standalone:

1. Presence detection, vital signs, fall detection — all on-device (Tier 2)
2. Gesture recognition, anomaly detection, zone tracking — via WASM modules (Tier 3)
3. Self-healing mesh — nodes detect when a neighbor goes down and redistribute coverage
4. Events emitted locally — can trigger GPIO, buzzer, or LED without any server

The server is optional for visualization and aggregation, but the ESP32 is capable of independent operation.

Roadmap

ADR-041 defines 60 total modules. The 29 remaining include medical (sleep apnea, seizure detection, gait analysis), retail (queue length, customer flow), industrial (forklift proximity, confined space), and more exotic ones (sleep staging, emotion detection). These follow the same pattern — write no_std Rust, compile to wasm32, upload to ESP32.

See issue #111 for the full module list and install details.