-
Notifications
You must be signed in to change notification settings - Fork 5
Description
(Issue created from Teams discussion)
Issue
FlatConnect.boxes is currently empty. The last known date that we have data was 2/28/26
Investigations thus far
The recent prod push on February 25th included changes to remove a "box data converter"
The schema changed in late February: https://console.cloud.google.com/bigquery?ws=!1m7!1m6!12m5!1m3!1snih-nci-dceg-connect-prod-6d04!2sus-central1!3sc72f615c-5a5b-4e98-85a9-ac0e9492ef1c!2e1
Additional information on the potential root cause:
On February 27, prod Connect data contained bags with populated d_234868461 arrays (tube data). On February 28, a full historical query of prod data shows bags_with_tubes = 0. Since the scheduled transformation performs UNNEST(tubesInBag), and UNNEST of empty arrays returns zero rows, the scheduled job correctly produced an empty result set and replaced FlatConnect.boxes with zero rows.
The root cause is the absence of tube data in prod as of February 28
Requested Outcome
The goal would be to move Warren's current flattening code out of BQ and have Eddie add this to his current Airflow flattening process as we have for all other FlatConnect tables
Metadata
Metadata
Labels
Type
Projects
Status