testRun after hooks not called when there is a test failure and stopOnFirstFail is true

[chris-armstrong]

What is your Scenario?

I'm running my test code and using the testRun.after hook functionality to run some instrumentation code after the test run is finished so that I can flush any pending spans.

What is the Current behavior?

I'm observing that my testRun.after hook function is not being called after there is a test failure and I have stopOnFirstFail: true in my configuration.

What is the Expected behavior?

I'd expect that testRun.after would be called regardless of test failure or the stopOnFirstFail setting value.

What is the public URL of the test page? (attach your complete example)

I'm just using https://www.abc.net.au as the test page is irrelevant to the incident.

What is your TestCafe test code?

Run with npx testcafe chrome test.js

test.js

import { Selector, t } from "testcafe";

fixture("Getting started").page("https://www.abc.net.au");

test("A simple test that fails", async () => {
        await t.expect(Selector("hello").innerText).eql("world");
});

test("Another test that is expected to fail", async () => {
        await t.expect(Selector("hello").innerText).eql("there");
});

Your complete configuration file

.testcaferc.js

module.exports = {
        hooks: {
                testRun: {
                        before: async () => console.log("[TEST RUN] BEFORE"),
                        after: async () => console.log("[TEST RUN] AFTER"),
                },
        },
        stopOnFirstFail: true,
};

Your complete test report

This is the behaviour I get when stopOnFirstFail is false:

➜  testcafe-example npx testcafe chrome ./test.js
 Running tests in:
 - Chrome 139.0.0.0 / Sequoia 15

 Getting started
[TEST RUN] BEFORE
 ✖ A simple test that fails

   1) Cannot obtain information about the node because the specified selector does not match any node in the DOM tree.

       > | Selector('hello')

      Browser: Chrome 139.0.0.0 / Sequoia 15

          1 |import { Selector, t } from "testcafe";
          2 |
          3 |fixture("Getting started").page("https://www.abc.net.au");
          4 |
          5 |test("A simple test that fails", async () => {
       >  6 |    await t.expect(Selector("hello").innerText).eql("world");
          7 |});
          8 |
          9 |test("Another test that is expected to fail", async () => {
         10 |    await t.expect(Selector("hello").innerText).eql("there");
         11 |});

         at <anonymous> (/private/tmp/testcafe-example/test.js:6:46)
         at asyncGeneratorStep (/private/tmp/testcafe-example/test.js:1:40)
         at _next (/private/tmp/testcafe-example/test.js:1:40)
         at <anonymous> (/private/tmp/testcafe-example/test.js:1:40)
         at <anonymous> (/private/tmp/testcafe-example/test.js:1:40)


[TEST RUN] AFTER
 ✖ Another test that is expected to fail

   1) Cannot obtain information about the node because the specified selector does not match any node in the DOM tree.

       > | Selector('hello')

      Browser: Chrome 139.0.0.0 / Sequoia 15

          5 |test("A simple test that fails", async () => {
          6 |    await t.expect(Selector("hello").innerText).eql("world");
          7 |});
          8 |
          9 |test("Another test that is expected to fail", async () => {
       > 10 |    await t.expect(Selector("hello").innerText).eql("there");
         11 |});
         12 |

         at <anonymous> (/private/tmp/testcafe-example/test.js:10:46)
         at asyncGeneratorStep (/private/tmp/testcafe-example/test.js:1:40)
         at _next (/private/tmp/testcafe-example/test.js:1:40)
         at <anonymous> (/private/tmp/testcafe-example/test.js:1:40)
         at <anonymous> (/private/tmp/testcafe-example/test.js:1:40)



 2/2 failed (27s)

When I enable it, you can see the [TEST RUN] AFTER statement is not printed.

➜  testcafe-example npx testcafe chrome ./test.js
 Running tests in:
 - Chrome 139.0.0.0 / Sequoia 15

 Getting started
[TEST RUN] BEFORE
 ✖ A simple test that fails

   1) Cannot obtain information about the node because the specified selector does not match any node in the DOM tree.

       > | Selector('hello')

      Browser: Chrome 139.0.0.0 / Sequoia 15

          1 |import { Selector, t } from "testcafe";
          2 |
          3 |fixture("Getting started").page("https://www.abc.net.au");
          4 |
          5 |test("A simple test that fails", async () => {
       >  6 |    await t.expect(Selector("hello").innerText).eql("world");
          7 |});
          8 |
          9 |test("Another test that is expected to fail", async () => {
         10 |    await t.expect(Selector("hello").innerText).eql("there");
         11 |});

         at <anonymous> (/private/tmp/testcafe-example/test.js:6:46)
         at asyncGeneratorStep (/private/tmp/testcafe-example/test.js:1:40)
         at _next (/private/tmp/testcafe-example/test.js:1:40)
         at <anonymous> (/private/tmp/testcafe-example/test.js:1:40)
         at <anonymous> (/private/tmp/testcafe-example/test.js:1:40)



 2/2 failed (12s)
➜  testcafe-example

Screenshots

No response

Steps to Reproduce

1. Run with the configuration and minimal test example above.
2. Set stopOnFirstFail to false to validate the normal behaviour.

TestCafe version

3.7.2

Node.js version

v22.11.0

Command-line arguments

testcafe chrome test.js

Browser name(s) and version(s)

Chrome 139.0.7258.139

Platform(s) and version(s)

Sequoia 15.6.1

Other

I've tried to debug this, but from what I can see, TestRunHookController.runTestRunAfterHookIfNecessary will not be called for each of the tests are skipped because of a previous test failure, such that this.pendingTestRunCount won't be decremented to zero to actually allow calling the after hook function.

    /* from `src/runner/test-run-hook-controller.js` */
    public async runTestRunAfterHookIfNecessary (testRun: TestRun): Promise<void> {
        this.pendingTestRunCount--;

        if (this.pendingTestRunCount === 0 && this.afterFn) {
            testRun.phase = TEST_RUN_PHASE.inTestRunAfterHook;

            try {
                await this.afterFn(this.testRunCtx);
            }
            catch (err) {
                testRun.addError(processTestFnError(err));
            }
        }
    }

I can see that my after hook is called when skipOnFirstFail is set to false, with the above function being called after each test (even if they fail).

[github-actions]

We appreciate you taking the time to share information about this issue. We reproduced the bug and added this ticket to our internal task queue. We'll update this thread once we have news.