OU-632: Split ALERTS query_range into several requests for Incidents

[rioloc]

This PR introduces the following changes:

ALERTS expression simplified

• The previous expression used as query param in the query_range endpoint is simplified as
  ALERTS{} . The second half of the query, + on () group_left (component, layer) (absent(meta{layer="${query.layer}", component="${query.component}"}))), was removed because component and layer labels are already present in the incident entity.

Split of query_range request into multiple ones for long queries

• When the query expression, composed by several ALERTS{} joined via or condition, exceeds a MAX_URL_LENGTH limit, then several queries will be created instead.
• The fetchDataForIncidentsAndAlerts was updated to support both single and multiple queries. In the case of multiple queries then multiple requests will be triggered to the Prometheus API.

skip alert if no incident is matched

• Added logic to skip the alert if no matchinIncident is found. This should be an edge case that could happen if an alert is triggered and a new incident metric was not produced yet.

Assisted-By: Claude Code

[openshift-ci-robot]

@rioloc: This pull request references OU-632 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.21.0" version, but no target version was set.

Details

In response to this:

This PR introduces the following changes:

ALERTS expression simplified

• The previous expression used as query param in the query_range endpoint is simplified as
  ALERTS{} . The second half of the query, + on () group_left (component, layer) (absent(meta{layer="${query.layer}", component="${query.component}"}))), was removed because component and layer labels are already present in the incident entity.

Split of query_range request into multiple ones for long queries

• When the query expression, composed by several ALERTS{} joined via or condition, exceeds a MAX_URL_LENGTH limit, then several queries will be created instead.
• The fetchDataForIncidentsAndAlerts was updated to support both single and multiple queries. In the case of multiple queries then multiple requests will be triggered to the Prometheus API.

Assisted-By: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[rioloc]

/retest

[openshift-ci-robot]

@rioloc: This pull request references OU-632 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR introduces the following changes:

ALERTS expression simplified

• The previous expression used as query param in the query_range endpoint is simplified as
  ALERTS{} . The second half of the query, + on () group_left (component, layer) (absent(meta{layer="${query.layer}", component="${query.component}"}))), was removed because component and layer labels are already present in the incident entity.

Split of query_range request into multiple ones for long queries

• When the query expression, composed by several ALERTS{} joined via or condition, exceeds a MAX_URL_LENGTH limit, then several queries will be created instead.
• The fetchDataForIncidentsAndAlerts was updated to support both single and multiple queries. In the case of multiple queries then multiple requests will be triggered to the Prometheus API.

skip alert if no incident is matched

Added logic to skip the alert if no matchinIncident is found. This should be an edge case that could happen if an alert is triggered and a new incident metric was not produced yet.

Assisted-By: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@rioloc: This pull request references OU-632 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR introduces the following changes:

ALERTS expression simplified

• The previous expression used as query param in the query_range endpoint is simplified as
  ALERTS{} . The second half of the query, + on () group_left (component, layer) (absent(meta{layer="${query.layer}", component="${query.component}"}))), was removed because component and layer labels are already present in the incident entity.

Split of query_range request into multiple ones for long queries

• When the query expression, composed by several ALERTS{} joined via or condition, exceeds a MAX_URL_LENGTH limit, then several queries will be created instead.
• The fetchDataForIncidentsAndAlerts was updated to support both single and multiple queries. In the case of multiple queries then multiple requests will be triggered to the Prometheus API.

skip alert if no incident is matched

• Added logic to skip the alert if no matchinIncident is found. This should be an edge case that could happen if an alert is triggered and a new incident metric was not produced yet.

Assisted-By: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[falox]

a couple of minor points

/lgtm

@DavidRajnoha @rioloc make sure to test what happens when one of the subqueries fails (expected: the parent call fails). Also, make sure that the splitting + merging doesn't affect the ordering of the data points (probably covered in the unit tests, but I comment anyway).

[rioloc]

make sure to test what happens when one of the subqueries fails (expected: the parent call fails).

If I well understood the point, that is guaranteed by the Promise.all which fails if any of the promises return an error.

[rioloc]

make sure that the splitting + merging doesn't affect the ordering of the data points (probably covered in the unit tests, but I comment anyway).

Also for this the order should be guaranteed to be deterministic, so the one of the results of customQuery | string[].
In 6a724e5 I've added another unit to check result ordering is correct

[DavidRajnoha]

/label qe-approved

[openshift-ci-robot]

@rioloc: This pull request references OU-632 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR introduces the following changes:

ALERTS expression simplified

• The previous expression used as query param in the query_range endpoint is simplified as
  ALERTS{} . The second half of the query, + on () group_left (component, layer) (absent(meta{layer="${query.layer}", component="${query.component}"}))), was removed because component and layer labels are already present in the incident entity.

Split of query_range request into multiple ones for long queries

• When the query expression, composed by several ALERTS{} joined via or condition, exceeds a MAX_URL_LENGTH limit, then several queries will be created instead.
• The fetchDataForIncidentsAndAlerts was updated to support both single and multiple queries. In the case of multiple queries then multiple requests will be triggered to the Prometheus API.

skip alert if no incident is matched

• Added logic to skip the alert if no matchinIncident is found. This should be an edge case that could happen if an alert is triggered and a new incident metric was not produced yet.

Assisted-By: Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[PeterYurkovich]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: falox, PeterYurkovich, rioloc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [PeterYurkovich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: falox, PeterYurkovich, rioloc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [PeterYurkovich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 4b9417f and 2 for PR HEAD d0a451c in total

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD d693b90 and 1 for PR HEAD d0a451c in total

[PeterYurkovich]

/override ci/prow/okd-scos-images

[openshift-ci]

@PeterYurkovich: Overrode contexts on behalf of PeterYurkovich: ci/prow/okd-scos-images

Details

In response to this:

/override ci/prow/okd-scos-images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[PeterYurkovich]

Currently okd image tests are down across the whole openshift org, overriding the test

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD a243f06 and 0 for PR HEAD d0a451c in total

[openshift-ci-robot]

/hold

Revision d0a451c was retested 3 times: holding

[jgbernalp]

/test e2e-monitoring

[openshift-ci]

@jgbernalp: /override requires failed status contexts, check run or a prowjob name to operate on.
The following unknown contexts/checkruns were given:

• okd-scos-images

Only the following failed contexts/checkruns were expected:

• ci/prow/e2e-aws-ovn
• ci/prow/e2e-monitoring
• ci/prow/images
• ci/prow/lint
• ci/prow/okd-scos-images
• ci/prow/periodics-images
• ci/prow/translations
• ci/prow/verify-deps
• pull-ci-openshift-monitoring-plugin-main-e2e-aws-ovn
• pull-ci-openshift-monitoring-plugin-main-e2e-monitoring
• pull-ci-openshift-monitoring-plugin-main-images
• pull-ci-openshift-monitoring-plugin-main-lint
• pull-ci-openshift-monitoring-plugin-main-okd-scos-images
• pull-ci-openshift-monitoring-plugin-main-periodics-images
• pull-ci-openshift-monitoring-plugin-main-translations
• pull-ci-openshift-monitoring-plugin-main-verify-deps
• tide

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

Details

In response to this:

/override okd-scos-images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jgbernalp]

/override ci/prow/okd-scos-images

[openshift-ci]

@jgbernalp: Overrode contexts on behalf of jgbernalp: ci/prow/okd-scos-images

Details

In response to this:

/override ci/prow/okd-scos-images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[jgbernalp]

/test e2e-monitoring

[openshift-ci]

@rioloc: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-monitoring	d0a451c	link	false	/test e2e-monitoring

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.