Run mutation tests in CI for changed files by souradeep-das · Pull Request #1364 · ethereum/execution-specs

souradeep-das · 2025-08-08T15:07:12Z

What was wrong?

Related to Issue #1362 and #1145

How was it fixed?

Run mutation tests on files affected in the PR (files are the atomic unit for running tests, so if a file was changed all of its mutations will be tested, mutmut currently does not allow function specific tests - but a patch can be created)
Segregate by fork and run the test suite (tests/{fork_name}) for the files changed in that fork

Test run on fork: https://github.com/souradeep-das/execution-specs/actions/runs/17051785519

Note - needs #1361 to be merged

Cute Animal Picture

souradeep-das · 2025-08-08T15:07:50Z

.github/workflows/test.yaml

+        run: pip install mutmut==3.3.0
+      - name: Apply mutmut patch
+        run: |
+          git clone https://github.com/souradeep-das/mutate-execution-specs.git


should i copy the patch to this repo instead?

Instead of patching, you can install a git dependency:

pip install git+https://github.com/souradeep-das/mutmut.git@<commit hash>

I think that would be easier to maintain than a patch file.

On a separate note, would something like this work (and maybe we can upstream)?

diff --git a/mutmut/__main__.py b/mutmut/__main__.py index 0fb1ad9..15b16ef 100644 --- a/mutmut/__main__.py +++ b/mutmut/__main__.py @@ -407,6 +407,8 @@ class PytestRunner(TestRunner): if mutmut.config.debug: params = ['-vv'] + params print('python -m pytest ', ' '.join(params)) + if mutmut.config.pytest_extra_args: + params += mutmut.config.pytest_extra_args exit_code = int(pytest.main(params, **kwargs)) if mutmut.config.debug: print(' exit code', exit_code) @@ -676,6 +678,7 @@ class Config: debug: bool paths_to_mutate: List[Path] tests_dir: List[str] = None + pytest_extra_args: List[str] = None def should_ignore_for_mutation(self, path): if not str(path).endswith('.py'): @@ -756,6 +759,7 @@ def load_config(): for y in s('paths_to_mutate', []) ] or guess_paths_to_mutate(), tests_dir=s('tests_dir', []), + pytest_extra_args=s('pytest_extra_args', []), )

souradeep-das · 2025-08-08T15:08:28Z

.github/workflows/test.yaml

+    if: github.event_name == 'pull_request'
+    strategy:
+      matrix:
+        fork: [frontier, homestead, tangerine_whistle, spurious_dragon, byzantium, constantinople, istanbul, berlin, london, paris, shanghai, cancun, prague, osaka]


some forks do not have a direct tests/{fork_name} file - what should we do for them?

This is likely going to change with @gurukamath's recent work on refactoring the tests.

souradeep-das · 2025-08-08T15:12:26Z

.github/workflows/test.yaml

The mutation tests might take a while to run - so I'm wondering how we should run them - per PR or commit on main

If several files have changes the time to run on CI also might be unrealistic. In which case I was thinking if we should add more limitations on when to run these tests (or patch mutmut to be able to run mutation tests for specific functions (as opposed to files).
Attached a test run on the main comment ^

SamWilsn · 2025-08-08T21:25:48Z

.github/workflows/test.yaml

+        run: pip install mutmut==3.3.0
+      - name: Apply mutmut patch
+        run: |
+          git clone https://github.com/souradeep-das/mutate-execution-specs.git


Instead of patching, you can install a git dependency:

pip install git+https://github.com/souradeep-das/mutmut.git@<commit hash>

I think that would be easier to maintain than a patch file.

On a separate note, would something like this work (and maybe we can upstream)?

diff --git a/mutmut/__main__.py b/mutmut/__main__.py index 0fb1ad9..15b16ef 100644 --- a/mutmut/__main__.py +++ b/mutmut/__main__.py @@ -407,6 +407,8 @@ class PytestRunner(TestRunner): if mutmut.config.debug: params = ['-vv'] + params print('python -m pytest ', ' '.join(params)) + if mutmut.config.pytest_extra_args: + params += mutmut.config.pytest_extra_args exit_code = int(pytest.main(params, **kwargs)) if mutmut.config.debug: print(' exit code', exit_code) @@ -676,6 +678,7 @@ class Config: debug: bool paths_to_mutate: List[Path] tests_dir: List[str] = None + pytest_extra_args: List[str] = None def should_ignore_for_mutation(self, path): if not str(path).endswith('.py'): @@ -756,6 +759,7 @@ def load_config(): for y in s('paths_to_mutate', []) ] or guess_paths_to_mutate(), tests_dir=s('tests_dir', []), + pytest_extra_args=s('pytest_extra_args', []), )

SamWilsn · 2025-08-08T21:28:38Z

.github/workflows/test.yaml

+    if: github.event_name == 'pull_request'
+    strategy:
+      matrix:
+        fork: [frontier, homestead, tangerine_whistle, spurious_dragon, byzantium, constantinople, istanbul, berlin, london, paris, shanghai, cancun, prague, osaka]


This is likely going to change with @gurukamath's recent work on refactoring the tests.

SamWilsn · 2025-08-08T21:44:02Z

.github/workflows/test.yaml

+        id: changed
+        run: |
+          git fetch origin ${{ github.event.pull_request.base.ref }}
+          CHANGED_FILES=$(git diff --name-only origin/${{ github.event.pull_request.base.ref }} ${{ github.sha }} | grep "^src/ethereum/${{ matrix.fork }}/" || true)


Do you need to grep, or can you just do something like git diff ... -- src/ethereum/${{ matrix.fork }}/?

SamWilsn · 2025-08-08T21:46:12Z

.github/workflows/test.yaml

+          CHANGED_FILES=$(git diff --name-only origin/${{ github.event.pull_request.base.ref }} ${{ github.sha }} | grep "^src/ethereum/${{ matrix.fork }}/" || true)
+          echo "changed_files<<EOF" >> $GITHUB_OUTPUT
+          echo "$CHANGED_FILES" >> $GITHUB_OUTPUT
+          echo "EOF" >> $GITHUB_OUTPUT


Suggested change

CHANGED_FILES=$(git diff --name-only origin/${{ github.event.pull_request.base.ref }} ${{ github.sha }} | grep "^src/ethereum/${{ matrix.fork }}/" || true)

echo "changed_files<<EOF" >> $GITHUB_OUTPUT

echo "$CHANGED_FILES" >> $GITHUB_OUTPUT

echo "EOF" >> $GITHUB_OUTPUT

CHANGED_FILES=$()

echo "changed_files<<EOF" >> $GITHUB_OUTPUT

( git diff --name-only origin/${{ github.event.pull_request.base.ref }} ${{ github.sha }} | grep "^src/ethereum/${{ matrix.fork }}/" || true ) >> $GITHUB_OUTPUT

echo "EOF" >> $GITHUB_OUTPUT

Can avoid storing the list in a temporary variable.

SamWilsn · 2025-08-08T21:58:09Z

.github/workflows/test.yaml

+          echo '[tool.mutmut]' > pyproject.toml
+          FILES=$(echo "${{ steps.changed.outputs.changed_files }}" | awk '{printf "\"%s\", ", $0}' | sed 's/, $//')
+          echo "paths_to_mutate = [ $FILES ]" >> pyproject.toml
+          echo "tests_dir = [\"tests/${{ matrix.fork }}/\"]" >> pyproject.toml
+          echo "debug = true" >> pyproject.toml


Hm, this is... unusual.

So first, this might be easier to implement with shell: python instead of bash. tomllib is in the standard library.

Second, we shouldn't need to modify our project configuration at all to get this to work. I think it would make sense to try and get a patch into mutmut that allows passing pytest arguments via the command line (unlike my patch above) or maybe via an environment variable. We can't be the only project with stupidly long running tests.

On a separate note, would something like this work (and maybe we can upstream)?

Is the patch you mentioned above, for any specific cases that we have in mind? or is it just intended to supply in options like test_dir?

I updated mutmut to take in options through the cli - and its on a PR upstream

also looks like mutmut is not particularly leaning towards enabling the cli options

boxed/mutmut#422 (comment)

SamWilsn · 2025-08-08T22:00:56Z

.github/workflows/test.yaml

        run: tox -e optimized
+
+  mutation-test:
+    runs-on: ubuntu-latest


Suggested change

runs-on: ubuntu-latest

runs-on: [self-hosted-ghr, size-xl-x64]

This is the magic sauce that runs the job on a beefier machine.

SamWilsn · 2025-08-08T22:06:23Z

.github/workflows/test.yaml

+    if: github.event_name == 'pull_request'
+    strategy:
+      matrix:
+        fork: [frontier, homestead, tangerine_whistle, spurious_dragon, byzantium, constantinople, istanbul, berlin, london, paris, shanghai, cancun, prague, osaka]


If you generate the matrix from the output of a previous job (i.e. ${{ needs.changed-forks.outputs.changed-forks }} where changed-forks is a preceding job that stores the changed forks), you can cut down own the number of runs plus use ethereum_spec_tools.forks:Hardfork to dynamically generate the list instead of having to hard code it.

thanks - that sounds great!

souradeep-das added 2 commits August 8, 2025 16:57

run mutmut on changed files

349e909

style fix

5942527

souradeep-das commented Aug 8, 2025

View reviewed changes

SamWilsn reviewed Aug 8, 2025

View reviewed changes

fix review comments

6cdcdaa

souradeep-das requested a review from SamWilsn August 14, 2025 07:20

update test run

3ff814d

danceratopz mentioned this pull request Aug 25, 2025

chore(tooling): experiment with mutmut ethereum/execution-spec-tests#1445

Closed

danceratopz pushed a commit to danceratopz/execution-specs that referenced this pull request Oct 22, 2025

fix(github): Pin cmake to a pre-4.0.0 version (ethereum#1364)

e4793e6

SamWilsn added this to the Post-Osaka Refactoring milestone Dec 15, 2025

	runs-on: ubuntu-latest
	runs-on: [self-hosted-ghr, size-xl-x64]

Comments

Conversation

souradeep-das commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong?

How was it fixed?

Cute Animal Picture

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SamWilsn Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

souradeep-das commented Aug 8, 2025 •

edited

Loading

SamWilsn Aug 8, 2025 •

edited

Loading