Skip to content

feat(ci): use pytest-split for optimal test distribution#1141

Merged
vringar merged 1 commit intomasterfrom
ci/pytest-split
Feb 26, 2026
Merged

feat(ci): use pytest-split for optimal test distribution#1141
vringar merged 1 commit intomasterfrom
ci/pytest-split

Conversation

@vringar
Copy link
Contributor

@vringar vringar commented Feb 23, 2026

Summary

Replaces the alphabetical glob-based test splitting (test/test_[a-e]*, etc.) with pytest-split for duration-based optimal distribution across CI runners.

  • Adds pytest-split dependency and a .test_durations file with real timing data from CI
  • Updates workflow matrix from glob patterns to numbered groups (1-7)
  • scripts/ci.sh conditionally uses --splits/--group in CI, falls back to $TESTS for local use

Current bottleneck: test/test_[p]* (mostly test_profile.py) takes ~31.9 min while other groups finish in 3-12 min.

Expected improvement: With optimal splitting, max runner time drops from ~31.9 min to ~9.5 min (3.4x speedup).

Supersedes #1103 which attempted the same goal with a custom bin-packing script and static distribution file. pytest-split is a well-maintained plugin that handles new/renamed tests gracefully (assigns default durations) and uses a proven least_duration splitting algorithm.

Test plan

  • CI passes with all 7 test groups running via pytest-split
  • Verify test groups are roughly balanced in duration (check CI job times)
  • Confirm coverage upload still works (artifact names changed from strategy.job-index to matrix.group)

@codecov
Copy link

codecov bot commented Feb 23, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.64%. Comparing base (d327d69) to head (2a368f5).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1141   +/-   ##
=======================================
  Coverage   56.64%   56.64%           
=======================================
  Files          40       40           
  Lines        3910     3910           
=======================================
  Hits         2215     2215           
  Misses       1695     1695           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vringar vringar force-pushed the ci/pytest-split branch 2 times, most recently from 7f1bda6 to 1eba960 Compare February 23, 2026 22:59
@vringar vringar marked this pull request as ready for review February 26, 2026 23:05
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves CI test runtime by replacing static glob-based splitting with pytest-split, using historical per-test durations to balance 7 parallel test groups more evenly.

Changes:

  • Add pytest-split to the project environment and introduce a .test_durations dataset for duration-based scheduling.
  • Update CI to run tests as 7 numbered groups (1–7) rather than filename glob buckets.
  • Adjust scripts/ci.sh to use pytest-split when GROUP/SPLITS are set, while keeping a $TESTS-based local fallback.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
scripts/ci.sh Switches between CI duration-based splitting and local $TESTS execution.
environment.yaml Adds pytest-split dependency needed by CI.
.test_durations Provides real timing data for pytest-split’s least_duration algorithm.
.github/workflows/run-tests.yaml Replaces glob matrix with numeric groups and updates coverage artifact naming + failure summary step.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +58 to +60
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
f.write(f"### Group ${{ matrix.group }}: {bad}/{total} test(s) failed\n\n")
for tc in root.iter("testcase"):
Copy link

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this embedded Python script, the group label is written as ${{ matrix.group }} inside a Python f-string. GitHub expressions are not evaluated inside the runtime script (and {{/}} are treated as literal braces by Python), so the step summary will show a literal ${ matrix.group } instead of the actual group number. Pass the matrix value into the step via env: (or substitute it in bash) and use that value when writing the summary.

Copilot uses AI. Check for mistakes.
@vringar vringar merged commit 40e044f into master Feb 26, 2026
17 checks passed
@vringar vringar deleted the ci/pytest-split branch February 26, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants