KAFKA-18369: State updater's *-ratio metrics are incorrect #21201

lucliu1108 · 2025-12-21T21:42:09Z

Ticket: https://issues.apache.org/jira/browse/KAFKA-18369

Summary

This PR revises 4 DefaultStateUpdate metrics for Kafka Streams:

idle-ratio
active-restore-ratio
standby-update-ratio
checkpoint-ratio
These metrics should represent the ratio of time spent on a certain
action over a windowed duration, rather than an average of time spent
ratio over iterations. This is implemented using a windowedSum
aggregation for each of the 4 metrics, similar to
KAFKA-18615: StreamThread *-ratio metrics suffer from sampling bias #21160

Each rolling window is controlled by 2 configs:

metrics.sample.window.ms: The window of time a metrics sample is
computed over, default as 30 seconds.
metrics.num.samples: The number of samples maintained to compute
metrics, default as 2.

Reviewers: @mjsax

Pankraz76

kind of dry.

Pankraz76 · 2025-12-22T09:53:12Z

...s/src/main/java/org/apache/kafka/streams/processor/internals/metrics/StreamsMetricsImpl.java

    public static final String LATENCY_DESCRIPTION_SUFFIX = " in milliseconds";
    public static final String RATE_DESCRIPTION_PREFIX = "The average number of ";
    public static final String RATE_DESCRIPTION_SUFFIX = " per second";
+    public static final String WINDOWED_RATIO_DESCRIPTION_PREFIX = "The ratio, over a rolling measurement window, ";


Suggested change

public static final String WINDOWED_RATIO_DESCRIPTION_PREFIX = "The ratio, over a rolling measurement window, ";

public static final String WINDOWED_RATIO_DESCRIPTION_PREFIX = "The ratio, over a rolling measurement window, of the time this thread spent ";

is it only used like this?

Right now this prefix consts is only used for thread. But it could be used for describing against something other units in the future.
The name of this const doesn't carry meaning about the entity (e.g.time, thread) described, so I added another const here to address that.

Good, it’s your call, but I’d keep it simple and let whoever comes next make further adjustments if they’re ever needed.

Pankraz76 · 2025-12-22T09:53:24Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/ThreadMetrics.java

    private static final String PUNCTUATE_MAX_LATENCY_DESCRIPTION = "The maximum punctuate latency";
    private static final String PROCESS_RATIO_DESCRIPTION =
-        "The ratio, over a rolling measurement window, of the time this thread spent " +
+        WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +


Suggested change

WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +

WINDOWED_RATIO_DESCRIPTION_PREFIX +

Pankraz76 · 2025-12-22T09:53:33Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/ThreadMetrics.java

            "processing active tasks to the total elapsed time in that window.";
    private static final String PUNCTUATE_RATIO_DESCRIPTION =
-        "The ratio, over a rolling measurement window, of the time this thread spent " +
+        WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +


Suggested change

WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +

WINDOWED_RATIO_DESCRIPTION_PREFIX +

dry

Pankraz76 · 2025-12-22T09:53:42Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/ThreadMetrics.java

            "punctuating active tasks to the total elapsed time in that window.";
    private static final String POLL_RATIO_DESCRIPTION =
-        "The ratio, over a rolling measurement window, of the time this thread spent " +
+        WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +


Suggested change

WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +

WINDOWED_RATIO_DESCRIPTION_PREFIX +

dry

Pankraz76 · 2025-12-22T09:53:53Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/ThreadMetrics.java

            "polling records from the consumer to the total elapsed time in that window.";
    private static final String COMMIT_RATIO_DESCRIPTION =
-        "The ratio, over a rolling measurement window, of the time this thread spent " +
+        WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +


Suggested change

WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +

WINDOWED_RATIO_DESCRIPTION_PREFIX +

dry.

Pankraz76 · 2025-12-22T09:59:00Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStateUpdater.java

-        private static final String UPDATE_RATIO_DESCRIPTION = RATIO_DESCRIPTION + "updating standby tasks";
-        private static final String CHECKPOINT_RATIO_DESCRIPTION = RATIO_DESCRIPTION + "checkpointing tasks restored progress";
+        private static final String IDLE_RATIO_DESCRIPTION = WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spend being idle";
+        private static final String RESTORE_RATIO_DESCRIPTION = WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spend restoring active tasks";


Suggested change

private static final String RESTORE_RATIO_DESCRIPTION = WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spend restoring active tasks";

private static final String RESTORE_RATIO_DESCRIPTION = WINDOWED_RATIO_DESCRIPTION_PREFIX + " restoring active tasks";

not sure about this one

spent

spend

idk if this is any good, all I know this string occurs 22 times making it kind of dry.

"of the time this thread spen"

aliehsaeedii · 2026-01-13T12:35:54Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStateUpdater.java

-        private static final String RESTORE_RATIO_DESCRIPTION = RATIO_DESCRIPTION + "restoring active tasks";
-        private static final String UPDATE_RATIO_DESCRIPTION = RATIO_DESCRIPTION + "updating standby tasks";
-        private static final String CHECKPOINT_RATIO_DESCRIPTION = RATIO_DESCRIPTION + "checkpointing tasks restored progress";
+        private static final String IDLE_RATIO_DESCRIPTION = WINDOWED_RATIO_DESCRIPTION_PREFIX +  THREAD_TIME_UNIT_DESCRIPTION + "being idle";


Any reason that we split a sentence into two vars (WINDOWED_RATIO_DESCRIPTION_PREFIX and THREAD_TIME_UNIT_DESCRIPTION)?

It's just syntax consideration. The RATIO_DESCRIPTION, like other PREFIX does not cover any actual unit (e.g. per xxx). But it's just my personal preference.

aliehsaeedii · 2026-01-13T12:38:56Z

...s/src/main/java/org/apache/kafka/streams/processor/internals/metrics/StreamsMetricsImpl.java

    public static final String LATENCY_DESCRIPTION_SUFFIX = " in milliseconds";
    public static final String RATE_DESCRIPTION_PREFIX = "The average number of ";
    public static final String RATE_DESCRIPTION_SUFFIX = " per second";
+    public static final String WINDOWED_RATIO_DESCRIPTION_PREFIX = "The ratio, over a rolling measurement window, ";


Good, it’s your call, but I’d keep it simple and let whoever comes next make further adjustments if they’re ever needed.

aliehsaeedii

Thanks @lucliu1108, I assume that we may need to announce the changes in docs/streams/upgrade-guide.html (same as #21160)

aliehsaeedii · 2026-01-13T13:10:26Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStateUpdater.java

+            final double activeRestoreTime = activeRestoreTimeWindowedSum.measure(metricsConfig, now);
+            final double standbyRestoreTime = standbyRestoreTimeWindowedSum.measure(metricsConfig, now);
+
+            final double totalTime = idleTime + checkpointTime + activeRestoreTime + standbyRestoreTime;


Does "total time" include the time the State Updater thread spends blocked on the internal queue or waiting for tasks? Is this time covered by idleTime?

Why don't we use the total time as totalTime = Math.max(0L, endTimeMs - totalStartTimeMs); as defined in runOnce()?

idleTime should cover the time blocked on the taskAndActions queue.

Good point, i will replace totalTime with the definition in runOnce().

aliehsaeedii · 2026-01-14T13:32:42Z

Thanks @lucliu1108, I assume that we may need to announce the changes in docs/streams/upgrade-guide.html (same as #21160)

what about this one @lucliu1108

aliehsaeedii · 2026-01-14T15:51:15Z

docs/streams/upgrade-guide.md


+## Streams API changes in 4.3.0
+
+The streams thread metrics `commit-ratio`, `process-ratio`, `punctuate-ratio`, and `poll-ratio` , along with streams state updater metrics `active-restore-ratio`, `standby-restore-ratio`, `idle-ratio`, and `checkpoint-ratio` have been updated. Each metric now reports, over a rolling measurement window, the ratio of time this thread spends performing the given action (`{action}`) to the total elapsed time in that window. The effective window duration is determined by the metrics configuration: `metrics.sample.window.ms` (per-sample window length)and `metrics.num.samples` (number of rolling windows).


and poll-ratio ,

please remove the extra white space.

mjsax

Overall LGTM. Few nits.

docs/streams/upgrade-guide.md

mjsax · 2026-01-15T02:28:38Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStateUpdater.java

-            updaterMetrics.checkpointRatioSensor.record((double) totalCheckpointLatency / totalLatency, now);
+            recordWindowedSum(
+                now,
+                (double) totalWaitLatency,


Do we need to cast explicitly? The parameter of recordWindowedSum(...) is defined as double, so it seems the explicit cast is unnecessary? Same for the rows below?

streams/src/main/java/org/apache/kafka/streams/processor/internals/DefaultStateUpdater.java

Co-authored-by: Matthias J. Sax <mjsax@apache.org>

mjsax

Why do we not track "totalLatency" as a WindowedSum, too? It was done this way in #21160 as runOnceLatencyWindowedSum -- why do we not need the same approach in this PR?

change to windowed sum ratio metrics

d679df3

github-actions bot added triage PRs from the community streams labels Dec 21, 2025

update metric prefix for StreamThread

cd124a4

Pankraz76 reviewed Dec 22, 2025

View reviewed changes

convert string to const

04467dc

github-actions bot removed the triage PRs from the community label Dec 23, 2025

frankvicky added ci-approved and removed streams labels Jan 3, 2026

lucliu1108 added 2 commits January 9, 2026 11:14

Merge branch 'trunk' into KAFKA-18369

3d1ab86

adjust import order

b3887d3

github-actions bot added the streams label Jan 9, 2026

aliehsaeedii reviewed Jan 13, 2026

View reviewed changes

lucliu1108 added 2 commits January 13, 2026 10:45

revise total latency

9f3107b

Merge branch 'trunk' into KAFKA-18369

dcf8af0

lucliu1108 added 2 commits January 14, 2026 09:35

add to upgrade doc

b3d8b69

remove unnecessary spacing

ba28acd

aliehsaeedii reviewed Jan 14, 2026

View reviewed changes

aliehsaeedii approved these changes Jan 14, 2026

View reviewed changes

remove space

c8e6b5b

mjsax reviewed Jan 15, 2026

View reviewed changes

lucliu1108 and others added 2 commits January 15, 2026 16:27

Apply suggestions from code review

bcc1cf2

Co-authored-by: Matthias J. Sax <mjsax@apache.org>

remove explicit casting from long to double

e14379d

lucliu1108 requested a review from mjsax January 15, 2026 22:37

mjsax reviewed Jan 23, 2026

View reviewed changes

	public static final String WINDOWED_RATIO_DESCRIPTION_PREFIX = "The ratio, over a rolling measurement window, ";
	public static final String WINDOWED_RATIO_DESCRIPTION_PREFIX = "The ratio, over a rolling measurement window, of the time this thread spent ";

	WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spent " +
	WINDOWED_RATIO_DESCRIPTION_PREFIX +

	private static final String RESTORE_RATIO_DESCRIPTION = WINDOWED_RATIO_DESCRIPTION_PREFIX + "of the time this thread spend restoring active tasks";
	private static final String RESTORE_RATIO_DESCRIPTION = WINDOWED_RATIO_DESCRIPTION_PREFIX + " restoring active tasks";


		## Streams API changes in 4.3.0

		The streams thread metrics `commit-ratio`, `process-ratio`, `punctuate-ratio`, and `poll-ratio` , along with streams state updater metrics `active-restore-ratio`, `standby-restore-ratio`, `idle-ratio`, and `checkpoint-ratio` have been updated. Each metric now reports, over a rolling measurement window, the ratio of time this thread spends performing the given action (`{action}`) to the total elapsed time in that window. The effective window duration is determined by the metrics configuration: `metrics.sample.window.ms` (per-sample window length)and `metrics.num.samples` (number of rolling windows).

KAFKA-18369: State updater's *-ratio metrics are incorrect #21201

Are you sure you want to change the base?

KAFKA-18369: State updater's *-ratio metrics are incorrect #21201

Conversation

lucliu1108 commented Dec 21, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Pankraz76 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aliehsaeedii left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aliehsaeedii commented Jan 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjsax left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mjsax left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

lucliu1108 commented Dec 21, 2025 •

edited by github-actions bot

Loading

aliehsaeedii left a comment •

edited

Loading

mjsax left a comment •

edited

Loading