Can I prevent mutants which mypy kills?

[MartinThoma]

I have many "false" surviving mutants like this one:

This mutant would be killed by mypy in the CI pipeline. However, it survives as I only run pytest -m "not slow" -x and pytest -m "not slow " && mypy app/ does not work. I've also tried pytest -m "not slow" --mypy -x, but that runs mypy over the tests as well which fails / makes each run pretty slow.

Do you have any suggestions what I could do?

[boxed]

I guess this goes back to the idea of having names of mutation types and then being able to config to skip some of them. At this point you can use the advanced whitelisting system to just check for : bool in the line I guess. It's ugly but it would work.

Thinking about this a bit more.. maybe mutmut should be able to ship with two sets of configs: strict and loose. We really want to use strict mode in some libs, like tri.struct, tri.declarative and iommi. But maybe that shouldn't be the default as the mutations you have above aren't really low hanging fruit.

[pedernot]

I had this exact issue. I solved it by creating a bash script that runs both pytest and mypy and then passed that to mutmut. Something like

#!/bin/bash -e
mypy app/
pytest -x -m "not slow"

I ran mypy first, since for me that was significantly faster than the test suite, so it would fail faster if the mutant was caught by mypy.

[MartinThoma]

@pedernot I like this idea - maybe I could also like this order my tests by speed / "likelihood" that something goes wrong. And also have pytest fail directly after the first fail and not execute the whole test suite.

[boxed]

@pedernot you should absolutely add -x to the pytest call. There is no need to run any more tests if a test has already failed.

[pedernot]

Even better, thanks.

[MartinThoma]

@boxed I think I've heard you mention this on a podcast recently. I need to remember that one in future 😄 (btw: Thank you for talking about it in a podcast / putting that much time and effort in Open Source projects ❤️ )

[MartinThoma]

To summarize this:

1. No, it is not possible at the moment to prevent mutmut from creating mutants which violate types.
2. It is unclear if the logic required to do that is worth implementing, as mypy is pretty quick (but maybe one could add an extension for mutmut which automatically runs mypy just on the mutated file?)
3. For now, having a small shell script as shown by @pedernot is a good and working solution.

[MartinThoma]

@pedernot Could you please add the -x to the pytest call of your shell script above? I want people who come here and blindly copy&paste have something reasonable.

[boxed]

I've edited his comment to include -x.

[felixdivo]

Should this be added to the configuration documentation? I was really puzzled to see the tool suggest such mutations and think that it should use type checking do this by default for typed projects (that's my subjective point of view, I guess).

This was one of my specific problems:

-        spawned: List[Gaussian] = []  # okay, original
+        spawned: List[Gaussian] = None  # violates type List, mutations

Also, one could add other interesting mutations when types are inspected: Given x: Iterable[str] = [], one could mutate the right hand side to tuple() or dict() or even str().

[boxed]

@felixdivo I will gladly accept a PR to do so.

[felixdivo]

@felixdivo I will gladly accept a PR to do so.

Unfortunately, I won't have time to implement that.

[jtjeferreira]

Sorry to reused this old thread, but the workaround described here is no longer possible, right?

[Otto-AA]

Hi, @jtjeferreira . No, this workaround is not possible. Also, I think with the current mutmut implementation (which only runs a subset of tests per mutant) this would not necessarily be faster anymore.

I've created an issue #467 to track ideas on how we can use type hints in mutmut.