KAFKA-18615: StreamThread *-ratio metrics suffer from sampling bias

[aliehsaeedii]

This PR implements KAFKA-18615 by adding a windowSum aggregation that
computes the sum of values over a time window, so that commit-ratio,
poll-ratio, process-ratio, and punctuate-ratio represent the ratio
of the {action} over a window duration rather than a single iteration.

The effective window duration is whatever you configure for metrics:

• metrics.sample.window.ms (per-sample window length)
• times metrics.num.samples (number of rolling windows)

With the default Kafka metrics config, that is typically:
metrics.sample.window.ms = 30000 ms metrics.num.samples = 2 → ~60
seconds total rolling window.

Reviewers: Matthias J. Sax matthias@confluent.io, Bill Bejeck
bill@confluent.io, Vincent Potuček (@Pankraz76)

[mjsax]

Should we also update the description of each metric, saying it's windowed (eg in ThreadMetrics.java)

[bbejeck]

Thanks @aliehsaeedii - overall LGTM - can we add a description to docs/upgrade.html section for 4.2?

[mjsax]

Overall LGTM. Few more minor things.

[Pankraz76]

Suggested change

	final double latencyWindow =
	windowedSum.measure(metricsConfig, now);
	ratioSensor.record(latencyWindow / runOnceLatencyWindow);
	ratioSensor.record(windowedSum.measure(metricsConfig, now) / runOnceLatencyWindow);

[mjsax]

LGTM.

[bbejeck]

Thanks @aliehsaeedii LGTM, with one minor comment

[mjsax]

This fix wont' get into AK 4.2, but only AK 4.3. We are passed code-freeze and only blockers can get merged now.

We need to start a new section for 4.3 and move this change there.

[mjsax]

Thanks for the fix. Merged to trunk.

[Pankraz76]

Thanks for the fix. Merged to trunk.

Just for my information: how are we planning to merge this, given the mandatory code freeze around this time of year?

This is only for development and will not be released anytime soon, correct?

However, if a hotfix were required, it would no longer be possible due to the merged feature.

I may have understood the code freeze policy differently, so I would appreciate some clarification.

Thanks.

[mjsax]

Code freeze only applies to 4.2 branch, which is the release branch fro 4.2.0. We can always merge to trunk, and this fix (and everything else that is merged to trunk but not cherry-picked to 4.2 branch) will only go into 4.3.0 release.

[Pankraz76]

Thanks for the nifty update.

So we could also consider other fixes, like:

• MINOR: Add SanityCheck #21165 [S3599]  #21168
• MAJOR: Hotfix broken spotless config, adhere SSOT #21165 #21161 #21168 #21171