Skip to content

Conversation

@RoniKishner
Copy link

Description

This PR fixes the double-counting of subtest failures in pytest's final test summary. Previously, when a test with subtests had failures, both the subtest failure and the top-level test failure were counted as regular "failed" tests, leading to inflated failure counts.

Changes

  • Modified pytest_report_teststatus hook in src/_pytest/subtests.py to return "subtests failed" category instead of "failed" for subtest failures
  • Updated _determine_main_color in src/_pytest/terminal.py to consider "subtests failed" when determining the main color (red)
  • Enhanced parse_summary_nouns in src/_pytest/pytester.py:
    • Updated regex to handle multi-word outcomes like "subtests failed"
    • Added pluralization mapping for subtest outcomes
    • Added whitespace handling for parsed outcomes
  • Updated test expectations in testing/test_subtests.py to match the new summary format

Example

Before:
===================== 2 failed, 2 subtests passed in 0.02s =====================
After:
===================== 1 failed, 2 subtests passed, 1 subtests failed in 0.02s =====================

This provides clear visibility into:

  • How many top-level tests failed (1 failed)
  • How many subtests failed (1 subtests failed)
  • How many subtests passed (2 subtests passed)

Closes #13986

@psf-chronographer psf-chronographer bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Nov 20, 2025
@RoniKishner RoniKishner force-pushed the subtests-failure-info branch from 80ecce2 to 7fa74de Compare November 20, 2025 19:12
Copy link
Member

@nicoddemus nicoddemus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @RoniKishner for the PR.

I'm OK with separating subtest failures from normal failures.

I left a suggestion regarding the changelog, could you also take a look at the failing tests?

@RoniKishner RoniKishner force-pushed the subtests-failure-info branch 2 times, most recently from 4fc2e2e to 73aae92 Compare November 30, 2025 09:40
@RoniKishner RoniKishner force-pushed the subtests-failure-info branch 3 times, most recently from 8657e00 to 2e47b21 Compare November 30, 2025 14:17
@RoniKishner RoniKishner force-pushed the subtests-failure-info branch from 2e47b21 to 51f5cb5 Compare December 8, 2025 15:25
Co-authored-by: pytest bot <pytestbot@users.noreply.github.com>
@RoniKishner RoniKishner force-pushed the subtests-failure-info branch from 51f5cb5 to 02206c2 Compare December 8, 2025 15:47
@nicoddemus
Copy link
Member

@RoniKishner the tests are still failing, do you need help with that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix double-counting of subtest failures in test summary

2 participants