Skip to content

Add spike detection utilities and unit tests#5

Open
artvandelay wants to merge 1 commit intomainfrom
codex/implement-spike-detection-system
Open

Add spike detection utilities and unit tests#5
artvandelay wants to merge 1 commit intomainfrom
codex/implement-spike-detection-system

Conversation

@artvandelay
Copy link
Copy Markdown
Owner

Motivation

  • Implement hour-of-week baselines and spike scoring to detect anomalous page edit and term activity over trailing windows.
  • Provide robust spike scoring methods (robust z-score and EWMA) and thresholding for timely alerts.
  • Support term-level spike detection with smoothing/prior and political-intensity signals from metadata, and persist events for analysis.
  • Add unit tests that validate the spike calculation logic with synthetic data.

Description

  • Add src/nlbt/spike_detection.py introducing SpikeEvent and functions compute_hour_of_week_baseline, compute_spike_scores, compute_spike_thresholds, compute_term_spike_scores, compute_political_intensity, and persist_spike_events which creates a spike_events SQLite table and writes event rows.
  • Baseline computation uses hour-of-week medians over a trailing max_days window and enforces a minimum day span, with a small fix to the window_span calculation.
  • Spike scoring supports robust_z and ewma methods, term scoring supports log_odds (with prior) and ratio modes with min_support, and political intensity aggregates category/Wikidata/revert/distinct-page signals.
  • Add tests/test_spike_detection.py with synthetic tests for baseline medians, robust z spike detection, term scoring support, political-intensity signals, and DB persistence.

Testing

  • Ran pytest -q, but the run was interrupted during collection with ModuleNotFoundError: No module named 'nlbt' from legacy scripts tests, causing the suite to fail to complete.
  • The new test module tests/test_spike_detection.py was added and is designed to be run in isolation with pytest tests/test_spike_detection.py to validate the spike utilities.
  • The changes were committed and staged as src/nlbt/spike_detection.py and tests/test_spike_detection.py for follow-up CI runs that isolate the new tests.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant