Skip to content

[ENHANCEMENT] Check if WPT metric exists for feature before trying to insert #1890

@jcscottiii

Description

@jcscottiii

When the WPT Consumer job was first created, we told it to look at the last 2 years of runs every time the job. This was because the IDs were still being added and we wanted the job to backfill.

This leads to really long run times. Almost a full day.

This may contribute to the high spikes in traffic to WPT too.


Possible solutions

We should probably add a capstone table similar to what we do for the UMA export job.

-- DailyChromiumHistogramEnumCapstones records completed enum ingestions for a particular day.
CREATE TABLE IF NOT EXISTS DailyChromiumHistogramEnumCapstones (
ChromiumHistogramEnumID STRING(36) NOT NULL,
Day DATE NOT NULL,
FOREIGN KEY (ChromiumHistogramEnumID) REFERENCES ChromiumHistogramEnums(ID) ON DELETE CASCADE
) PRIMARY KEY (ChromiumHistogramEnumID, Day);

The primary key would be a combination of the Web feature ID and the WPT Run ID.

Alternatively, we could look at the merging logic:

// Read the existing metric and merge the values.
var existingMetric SpannerWPTRunFeatureMetric
err = row.ToStruct(&existingMetric)
if err != nil {
return nil, err
}
// Only allow overriding of the test numbers.
existingMetric.TestPass = cmp.Or[*int64](metric.TestPass, existingMetric.TestPass, nil)
existingMetric.TotalTests = cmp.Or[*int64](metric.TotalTests, existingMetric.TotalTests, nil)
existingMetric.TestPassRate = getPassRate(existingMetric.TestPass, existingMetric.TotalTests)
// Allow subtest metrics to be reset to nil.
existingMetric.SubtestPass = metric.SubtestPass
existingMetric.TotalSubtests = metric.TotalSubtests
existingMetric.SubtestPassRate = getPassRate(existingMetric.SubtestPass, existingMetric.TotalSubtests)
// Allow feature run details to be reset
existingMetric.FeatureRunDetails = metric.FeatureRunDetails
m, err = spanner.InsertOrUpdateStruct(WPTRunFeatureMetricTable, existingMetric)
if err != nil {
return nil, errors.Join(ErrInternalQueryFailure, err)
}
}

If we detect there are no changes, we could emit no mutation. That could save us an insert.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions