perf: unified scheduler telemetry, backpressure, and hot-loop optimizations

[ChrisLundquist]

Summary

Overhauls the unified scheduler for reduced lock contention and smarter GPU/CPU work distribution, adds opt-in telemetry instrumentation, and optimizes critical LZ/FSE hot loops. Introduces a local perf-gate script for regression-guarding throughput and scheduler overhead.

Unified scheduler improvements (parallel.rs)

• Reduced queue-lock churn: Workers now prepare the next task before re-acquiring the queue lock, performing a single lock/unlock for both task retirement and next-task enqueue. Previously, intermediate-stage completions held the lock across two separate critical sections (retire + enqueue), causing convoy effects under high worker counts.
• GPU coordinator fairness reordering: The coordinator now processes StageN completions first, then Fused requests, then Stage0 last. Processing downstream continuations first reduces in-flight work and avoids starving completions when bursts arrive together.
• Backpressure-aware Auto routing: Auto mode for stage1 GPU entropy now checks a pressure score before routing to GPU. try_send(Full) increments pressure by 2, Disconnected by 1, and successful sends decrement by 1. When pressure exceeds workers * 2, new blocks are routed to CPU. Explicit Gpu/Cpu backend assignments bypass backpressure.
• Eliminated per-block Vec<u8> clones for GPU requests: GpuRequest::Stage0 and GpuRequest::Fused now carry only a block index — the coordinator reads input data directly from the shared blocks slice, avoiding allocation + copy per GPU-routed block.
• Wider sync channel: Channel depth changed from min(4, num_blocks) to min(num_blocks, workers*2).clamp(1, 16) to reduce transient try_send(Full) fallbacks without unbounded buffering.
• Localized worker stage continuations: Per-block continuation state stays local in worker threads to avoid queue round-trips between intermediate stages, with direct GPU handoff when routing stays on GPU.
• Extracted shared completion state machine: The duplicated worker/GPU completion logic is now a single shared helper, reducing maintenance surface.

Telemetry infrastructure

• UnifiedSchedulerStats: per-run timing for stage compute, queue wait, queue admin, GPU handoff, and try_send failure counters.
• Collection gated behind AtomicBool (opt-in via set_unified_scheduler_stats_enabled()) — zero overhead in production.
• SchedulerRunRecorder uses Drop to flush per-invocation local atomics into global stats.
• --print-scheduler-stats flag in profile example emits SCHEDULER_STATS and PROFILE_STATS machine-readable lines.

Hot-loop optimizations

• FSE spread_symbols: Eliminates intermediate Vec<usize> position buffer — walks the permutation cycle in-place.
• FSE build_encode_tables: Replaces skip/take loop with slice::fill() for range initialization.
• FSE fse_encode_internal: SoA layout — splits Vec<(u32, u32)> into separate bit_values: Vec<u32> + bit_counts: Vec<u8>, reducing memory bandwidth in the write pass.
• LZ77 find_best_match / find_top_k_matches: Raw pointer arithmetic (compare_bytes_ptr) to skip repeated slice construction in the inner match-finding loop. Local variable hoisting for prev/window_mask.
• LzSeq select_best_match: Deduplicates repeat-offset comparisons when offsets are equal (common at stream start, saves up to 2 redundant calls).
• simd.rs: New compare_bytes_ptr unsafe wrapper for callers with pre-validated pointer bounds.

Perf gate script

• scripts/perf-gate.sh: Runs a fixed matrix of pipeline/mode combinations, collects median throughput and scheduler overhead across N repeats, and compares against a baseline TSV.
• Configurable thresholds: --throughput-regression-pct (default 4%), --overhead-regression-abs (default 0.02).
• Supports --threads, --cpu-only, --update-baseline, and WebGPU auto-detection.

Docs

• Deletes superseded gpu-strategy.md and 6 stale feedback files
• Updates unified scheduler north-star plan
• Adds perf-validation exec plan

Test plan

• ./scripts/test.sh --quick — fmt, clippy, all tests pass
• ./scripts/test.sh --all — all feature combinations pass
• ./scripts/perf-gate.sh --update-baseline — establish baseline
• ./scripts/perf-gate.sh — verify no regressions from baseline
• Manual: ./examples/profile --pipeline lzr --print-scheduler-stats shows SCHEDULER_STATS and PROFILE_STATS lines
• WebGPU round-trip: test_lzr_backend_assignments_are_interchangeable covers cpu/cpu, gpu/cpu, cpu/gpu, auto/auto
• Backpressure unit tests: test_stage1_auto_backpressure_biases_to_cpu, test_stage1_backpressure_does_not_override_explicit_backend

🤖 Generated with Claude Code