Trt 2389 test summaries

[neisw]

Adds tests data type

Produces test_summaries.json

Need to review the importance of this if we are pivoting to openshift/origin#30604

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[coderabbitai]

Walkthrough

This change introduces support for fetching and analyzing aggregated test summary data by period. It adds a new data model, extends the CI data client with a BigQuery query method, and implements a new "tests" data type handler in the historical data analyzer with formatting and validation updates.

Changes

Cohort / File(s)	Summary
Core Data Model pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go	New public struct TestSummaryByPeriodRow with fields for release, test name, test counts, failure metrics, duration, and date range. Includes BigQuery tags for SQL result mapping.
Data Client Enhancements pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go, pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go, pkg/jobrunaggregator/jobrunaggregatorlib/util.go	Added ListTestSummaryByPeriod method to CIDataClient interface and ciDataClient implementation to query aggregated test summaries. Added retry wrapper with slowBackoff. Added bigQueryLabelValueTestSummaryByPeriod constant.
Analyzer Tests Support pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go, pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go, pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go	Introduced "tests" data type handler with runTestsDataType method to fetch and format test summaries. Updated validation to conditionally require --current and release flags based on data type. Added formatTestOutput helper to sort and JSON-marshal test results.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• BigQuery query construction in ci_data_client.go — verify aggregation logic, GROUP BY/HAVING clauses, and date range handling
• Conditional validation logic in cmd.go — ensure flag requirements are correctly enforced for different data types
• Output sorting and formatting in util.go (analyzer) — confirm sort order (Release ascending, then FailureCount descending, TestName ascending) and JSON marshaling
• Integration between analyzer and retrying client — verify retry wrapper follows established patterns and error handling is consistent

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[neisw]

Likely need more validation for when we change releases. Make sure we have at least 2 weeks of data, etc.

[neisw]

/hold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neisw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/jobrunaggregator/OWNERS [neisw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go (1)

25-52: New data-type=tests flow is cleanly separated; consider future configurability

The early branch in Run and runTestsDataType cleanly bypass the comparison logic for data-type=tests, fetching summaries once and writing them directly via formatTestOutput, without disturbing the alerts/disruptions path. The hardcoded suiteName, daysBack, and minTestCount are reasonable defaults, but if this grows beyond a single use case, promoting them to flags or config would improve long‑term flexibility.

Also applies to: 100-132

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting

📥 Commits

Reviewing files that changed from the base of the PR and between f0f4b3b and c5366f4.

📒 Files selected for processing (7)

• pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1 hunks)
• pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (2 hunks)
• pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1 hunks)
• pkg/jobrunaggregator/jobrunaggregatorlib/util.go (1 hunks)
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go (2 hunks)
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go (2 hunks)
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (1 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**

⚙️ CodeRabbit configuration file

-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.

Files:

• pkg/jobrunaggregator/jobrunaggregatorlib/util.go
• pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go
• pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go
• pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/analyzer.go
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go
• pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go

🧬 Code graph analysis (3)

pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

• TestSummaryByPeriodRow (9-20)

pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1)

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

• TestSummaryByPeriodRow (9-20)

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (2)

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

• TestSummaryByPeriodRow (9-20)

pkg/api/types.go (1)

• Release (452-466)

🔇 Additional comments (6)

pkg/jobrunaggregator/jobrunaggregatorlib/util.go (1)

39-58: BigQuery label addition is consistent and low‑risk

The new bigQueryLabelValueTestSummaryByPeriod constant matches existing naming/usage patterns for query labels; no issues from a maintainability or behavior standpoint.

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/cmd.go (1)

29-49: data-type=tests wiring and validation look coherent

Adding "tests" to supportedDataTypes and relaxing only the --current / --previous-release requirements (while still enforcing BigQuery auth/coords when --new is empty) matches the new tests path, which fetches directly from BigQuery without doing comparisons. No obvious correctness or UX regressions here.

Also applies to: 51-77

pkg/jobrunaggregator/jobrunhistoricaldataanalyzer/util.go (1)

170-185: Test-summary formatting helper is straightforward and consistent

The in-place sort by release, failure count (desc), then test name mirrors the query’s ordering and existing formatOutput patterns, and json.MarshalIndent keeps output stable for diffs. Behavior for empty input is defensive and safe.

pkg/jobrunaggregator/jobrunaggregatorlib/retrying_ci_data_client.go (1)

188-196: Retry wrapper mirrors existing client behavior

ListTestSummaryByPeriod on the retrying client follows the same retry.OnError(slowBackoff, isReadQuotaError, …) pattern as the other methods, so retry semantics stay consistent and localized to quota-style failures.

If you haven’t already, run a quick integration test hitting the real BigQuery-backed implementation to confirm retry behavior and error wrapping are as expected.

pkg/jobrunaggregator/jobrunaggregatorapi/types_row_test_summary.go (1)

1-20: Struct shape correctly matches the BigQuery query schema

TestSummaryByPeriodRow’s field names, types, and bigquery tags line up with the ListTestSummaryByPeriod SELECT clause (SUMs as int64, ratios/durations as float64, dates as civil.Date), so decoding should be straightforward.

Please confirm via a small end‑to‑end query (or unit test with a fake RowIterator) that BigQuery scanning into this struct works without field/tag mismatches.

pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go (1)

70-76: Test-summary query is well‑structured and consistent with existing CIDataClient patterns

The new ListTestSummaryByPeriod method is documented clearly on the interface and the implementation’s SQL matches the TestSummaryByPeriodRow schema (field aliases ↔ struct tags). The aggregation logic (SUM/SAFE_DIVIDE/AVG, window via CURRENT_DATE() and daysBack, HAVING SUM(test_count) > @min_test_count) and labeling (bigQueryLabelValueTestSummaryByPeriod) align with how other queries in this client are written, so it should be straightforward to operate and debug.

If possible, run this against a representative dataset to confirm:

• Parameter binding for days_back / min_test_count is accepted by BigQuery.
• The DATE_SUB(CURRENT_DATE(), INTERVAL @days_back DAY) filter yields the expected window.
• The decoded TestSummaryByPeriodRow values (including FailureRate and PeriodStart/End) match manual query results.

Also applies to: 1107-1173

[openshift-ci]

@neisw: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/breaking-changes	c5366f4	link	false	/test breaking-changes
ci/prow/lint	c5366f4	link	true	/test lint
ci/prow/images	c5366f4	link	true	/test images
ci/prow/unit	c5366f4	link	true	/test unit

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.