-
Couldn't load subscription status.
- Fork 35
Description
When the WPT Consumer job was first created, we told it to look at the last 2 years of runs every time the job. This was because the IDs were still being added and we wanted the job to backfill.
This leads to really long run times. Almost a full day.
This may contribute to the high spikes in traffic to WPT too.
Possible solutions
We should probably add a capstone table similar to what we do for the UMA export job.
webstatus.dev/infra/storage/spanner/migrations/000008.sql
Lines 60 to 65 in 43a2f84
| -- DailyChromiumHistogramEnumCapstones records completed enum ingestions for a particular day. | |
| CREATE TABLE IF NOT EXISTS DailyChromiumHistogramEnumCapstones ( | |
| ChromiumHistogramEnumID STRING(36) NOT NULL, | |
| Day DATE NOT NULL, | |
| FOREIGN KEY (ChromiumHistogramEnumID) REFERENCES ChromiumHistogramEnums(ID) ON DELETE CASCADE | |
| ) PRIMARY KEY (ChromiumHistogramEnumID, Day); |
The primary key would be a combination of the Web feature ID and the WPT Run ID.
Alternatively, we could look at the merging logic:
webstatus.dev/lib/gcpspanner/wpt_run_feature_metric.go
Lines 296 to 316 in 43a2f84
| // Read the existing metric and merge the values. | |
| var existingMetric SpannerWPTRunFeatureMetric | |
| err = row.ToStruct(&existingMetric) | |
| if err != nil { | |
| return nil, err | |
| } | |
| // Only allow overriding of the test numbers. | |
| existingMetric.TestPass = cmp.Or[*int64](metric.TestPass, existingMetric.TestPass, nil) | |
| existingMetric.TotalTests = cmp.Or[*int64](metric.TotalTests, existingMetric.TotalTests, nil) | |
| existingMetric.TestPassRate = getPassRate(existingMetric.TestPass, existingMetric.TotalTests) | |
| // Allow subtest metrics to be reset to nil. | |
| existingMetric.SubtestPass = metric.SubtestPass | |
| existingMetric.TotalSubtests = metric.TotalSubtests | |
| existingMetric.SubtestPassRate = getPassRate(existingMetric.SubtestPass, existingMetric.TotalSubtests) | |
| // Allow feature run details to be reset | |
| existingMetric.FeatureRunDetails = metric.FeatureRunDetails | |
| m, err = spanner.InsertOrUpdateStruct(WPTRunFeatureMetricTable, existingMetric) | |
| if err != nil { | |
| return nil, errors.Join(ErrInternalQueryFailure, err) | |
| } | |
| } |
If we detect there are no changes, we could emit no mutation. That could save us an insert.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status