Skip to content

Conversation

@AlbertoSH
Copy link

@AlbertoSH AlbertoSH commented Nov 12, 2025

TL;DR;
Fixes #1029
plus removes some more unneeded metrics

When deleting a canary, its status metric is not updated. Having two canaries deployed, both initialized, metrics endpoint exposes

# HELP flagger_canary_status Last canary analysis result
# TYPE flagger_canary_status gauge
flagger_canary_status{name="canary1",namespace="foo"} 1
flagger_canary_status{name="canary2",namespace="foo"} 1
# HELP flagger_canary_total Total number of canary object
# TYPE flagger_canary_total gauge
flagger_canary_total{namespace="foo"} 2

After deleting one, the metrics are updated to

# HELP flagger_canary_status Last canary analysis result
# TYPE flagger_canary_status gauge
flagger_canary_status{name="canary1",namespace="foo"} 1   # <--- this one was deleted, but not reflected here
flagger_canary_status{name="canary2",namespace="foo"} 1
# HELP flagger_canary_total Total number of canary object
# TYPE flagger_canary_total gauge
flagger_canary_total{namespace="foo"} 1

Deleting the metric feels more appropriate than marking it as deleted, as otherwise the amount of timeseries can grow a lot and, if it's deleted, it means that it's probably not needed anymore

Signed-off-by: AlbertoSH <alberto.sanz.herrero@gmail.com>
@AlbertoSH AlbertoSH force-pushed the fix-1029-remove-status-metric-when-canary-is-deleted branch from cb333dc to fbd46e9 Compare November 12, 2025 09:46
Signed-off-by: AlbertoSH <alberto.sanz.herrero@gmail.com>
@AlbertoSH AlbertoSH changed the title Remove flagger_canary_status metric after a canary is deleted Remove metrics metric after a canary is deleted Nov 12, 2025
Copy link
Member

@aryan9600 aryan9600 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleting metrics for terminated canaries would be a breaking change. instead lets modify the status gauge to be set to 3 when a canary is terminated. this way, all prometheus queries can add a flagger_canary_status!=4 to exclude deleted canaries

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metric gauge for status not updated when canary deleted

2 participants