Fix: Partition core lookup table #2227
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
cmd/hatchet-migrate/migrate/migrations/20250829173445_v1_0_41.sql
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Pull Request Overview
This PR implements table partitioning for the core lookup table (v1_lookup_table) to improve performance and enable efficient data management. The migration follows a standard pattern: create a new partitioned table, backfill data, and cut over to the new structure.
Key changes:
- Modifies the primary key to include
inserted_atfor partition compatibility - Creates a partitioned version of the lookup table with weekly range partitions
- Updates trigger functions to work with the new composite primary key
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| sql/schema/v1-core.sql | Updates table schema and trigger functions to use composite primary key (external_id, inserted_at) |
| cmd/hatchet-migrate/migrate/migrations/20250829173445_v1_0_41.sql | Implements the complete migration with partitioned table creation, data backfill, and cutover logic |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
cmd/hatchet-migrate/migrate/migrations/20250829173445_v1_0_41.sql
Outdated
Show resolved
Hide resolved
| SELECT '19700101' INTO startDateStr; | ||
| SELECT TO_CHAR(date_trunc('week', (NOW() - INTERVAL '8 days')::DATE), 'YYYYMMDD') INTO endDateStr; | ||
| SELECT LOWER(FORMAT('%s_%s', targetTableName, startDateStr)) INTO newTableName; |
There was a problem hiding this comment.
Just to confirm, anything older than 8 days goes into the same partition?
There was a problem hiding this comment.
yeah this was my version of "from the beginning of time until last week" just to make sure we catch everything
cmd/hatchet-migrate/migrate/migrations/20250829173445_v1_0_41.sql
Outdated
Show resolved
Hide resolved
fix: clean up migration feat: migration
mrkaye97
force-pushed
the
mk/partition-core-lookup-table
branch
from
891fd61 to
7efbfc6
Compare
Description
Adding a migration to partition the core lookup table, similar to the one for the OLAP one. Basic steps are:
It's a bit different than with Timescale since we need to create the partitions manually. I was thinking it'd make sense to just have one big partition for all the old data, which will all get dropped in one shot after a week or so, but we could also loop through and create many. Definitely makes me nervous to need the three steps in this one since it's really not easily rerunnable / reversible and it's in the OSS
Type of change