feat(ci): use pytest-split for optimal test distribution#1141
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1141 +/- ##
=======================================
Coverage 56.64% 56.64%
=======================================
Files 40 40
Lines 3910 3910
=======================================
Hits 2215 2215
Misses 1695 1695 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
7f1bda6 to
1eba960
Compare
1eba960 to
2a368f5
Compare
There was a problem hiding this comment.
Pull request overview
This PR improves CI test runtime by replacing static glob-based splitting with pytest-split, using historical per-test durations to balance 7 parallel test groups more evenly.
Changes:
- Add
pytest-splitto the project environment and introduce a.test_durationsdataset for duration-based scheduling. - Update CI to run tests as 7 numbered groups (1–7) rather than filename glob buckets.
- Adjust
scripts/ci.shto usepytest-splitwhenGROUP/SPLITSare set, while keeping a$TESTS-based local fallback.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
scripts/ci.sh |
Switches between CI duration-based splitting and local $TESTS execution. |
environment.yaml |
Adds pytest-split dependency needed by CI. |
.test_durations |
Provides real timing data for pytest-split’s least_duration algorithm. |
.github/workflows/run-tests.yaml |
Replaces glob matrix with numeric groups and updates coverage artifact naming + failure summary step. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f: | ||
| f.write(f"### Group ${{ matrix.group }}: {bad}/{total} test(s) failed\n\n") | ||
| for tc in root.iter("testcase"): |
There was a problem hiding this comment.
In this embedded Python script, the group label is written as ${{ matrix.group }} inside a Python f-string. GitHub expressions are not evaluated inside the runtime script (and {{/}} are treated as literal braces by Python), so the step summary will show a literal ${ matrix.group } instead of the actual group number. Pass the matrix value into the step via env: (or substitute it in bash) and use that value when writing the summary.
Summary
Replaces the alphabetical glob-based test splitting (
test/test_[a-e]*, etc.) with pytest-split for duration-based optimal distribution across CI runners.pytest-splitdependency and a.test_durationsfile with real timing data from CIscripts/ci.shconditionally uses--splits/--groupin CI, falls back to$TESTSfor local useCurrent bottleneck:
test/test_[p]*(mostlytest_profile.py) takes ~31.9 min while other groups finish in 3-12 min.Expected improvement: With optimal splitting, max runner time drops from ~31.9 min to ~9.5 min (3.4x speedup).
Supersedes #1103 which attempted the same goal with a custom bin-packing script and static distribution file. pytest-split is a well-maintained plugin that handles new/renamed tests gracefully (assigns default durations) and uses a proven
least_durationsplitting algorithm.Test plan
strategy.job-indextomatrix.group)