Developer effectiveness vs. toolchain effectiveness

[UrsLange]

Hello and thank you very much for the hard work you put into creating this framework. It is great to have this groundwork publicly available.

The thing I would like to start a discussion about is the part about effectiveness and the singals for it.

First of all I can see why the listed signals are important and why one would want to track them. However I am not certain, that the given examples really give you a picture about your developers effectiveness.

An anology I really like is that

effectiveness means doing the right things and efficiency means doing them right

Of course this is not a definition but it helps understand how to distinguish the two, I think. The definition (quickly googled) for effectiveness is

the degree to which something is successful in producing a desired result; success.

and for efficiency

effective operation as measured by a comparison of production with cost (as in energy, time, and money)

Now I would argue that a flaky test signals an ineffective test. Or at greater scale an ineffective pipeline. But in most cases the developer is on the receiving end of the flakyness. For the person running a pipeline and being confronted with flakyness, I'd say the efficiency is negatively impacted, not the effectiveness. This is because in most cases the fact that failure is caused by flakyness is obvious for the developer or at least soon discovered. So the usual measure is to rerun the pipeline, causing mainly a loss of time.

I would not go as far and say your derivation is wrong. I just feel like other signals would draw a better picture of "developer effectiveness". Flakyness or even crashing pipelines, to me are a signal for "toolchain effectiveness", because the toolchain is clearly not fulfilling it's intended purpose. Generally I'd say that ineffective toolchains lead to decreasing developer efficiency. I propose to seperate these two aspects of effectiveness.

For developer effectiveness, I would be interested in, how I can help my engineering teams making the right choices and thereby enable them to make effective contributions. Referring to above analogy: "How can I help my teams doing the right things?". This involves giving them knowledge about how their actions will affect business, making correct information easily available and maybe providing helpful assistance in their doing.
For example the developer writing a test would have the intention that this test does not turn out flaky. That could be helped by some sort of tool in their IDE that warns them of common pitfalls.
I'll try to provide some example signals but please bare with me as I am just making them up and have in no way put them to the test:

• Business value created per change
• Probability for a developer to find information that is correct and complete. Complete meaning that it fully resolves the question.
• Fraction of newly created tests that are not flaky or otherwise ineffective.

I am looking forward to your feedback.

[mkanat]

Hey! All really interesting points. I think the key concept that's missing in most (not all) of what you wrote above is that both of the productivity signals require a developer intention before you can measure anything.

You did make this point about flakiness, and you're right, there, but I think it's useful to be explicit about what the developer's intention is. I think that developers intend to write tests that provide accurate and useful knowledge. I'm sure I could refine that further, but that's my rapid response. So then your signals are something like how accurate and useful the tests are. That's effectiveness, essentially. Then your efficiency signals are how much time and effort it takes to accomplish that result.

The specifics of what you can actually measure will depend a lot on your development environment. I can imagine a beautiful environment where the metrics would be nearly perfect, and you would be able to statically (or at worst, dynamically) tell whether your tests covered all possible behaviors of your system, how much that cost in human time to accomplish, how rapidly test results came back to developers when they ran them, how much overhead your test infrastructure was adding, etc. If you don't have that mythical platform, then you would have to find metrics that are more proxies for it.

Without the developer intention being specified, it becomes very hard to think about metrics, or even the signals---they become vague and generalized, with difficult-to-understand value to the business. With the developer intention specified, they become specific and actionable, provided you also take into account all the other things the framework talks about, like audience, etc.