Fix: Partition core v1_dag_to_task
#2241
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| CREATE TABLE v1_dag_to_task_original ( | ||
| dag_id BIGINT NOT NULL, | ||
| dag_inserted_at TIMESTAMPTZ NOT NULL, | ||
| task_id BIGINT NOT NULL, | ||
| task_inserted_at TIMESTAMPTZ NOT NULL, | ||
| PRIMARY KEY (dag_id, dag_inserted_at, task_id, task_inserted_at) | ||
| ); | ||
|
|
||
| INSERT INTO v1_dag_to_task_original | ||
| SELECT * FROM v1_dag_to_task; | ||
|
|
||
| DROP TABLE v1_dag_to_task; | ||
| ALTER TABLE v1_dag_to_task_original RENAME TO v1_dag_to_task; | ||
| ALTER INDEX v1_dag_to_task_original_pkey RENAME TO v1_dag_to_task_pkey; |
There was a problem hiding this comment.
this is mostly just for me testing on local
There was a problem hiding this comment.
Pull Request Overview
This PR partitions the v1_dag_to_task table by range on the dag_inserted_at column to improve performance for time-based queries. The migration creates a new partitioned table, preserves existing data as the first partition, and updates the schema definition.
- Converts
v1_dag_to_taskfrom a regular table to a partitioned table usingdag_inserted_atas the partition key - Creates a migration that preserves all existing data by making it the first partition
- Updates the schema definition to reflect the new partitioned structure
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| sql/schema/v1-core.sql | Updates table definition to add PARTITION BY RANGE(dag_inserted_at) |
| cmd/hatchet-migrate/migrate/migrations/20250903152758_v1_0_41.sql | Migration script to convert existing table to partitioned format with data preservation |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
@abelanger5 any idea how long |
| SELECT create_v1_range_partition('v1_dag_to_task_partitioned', NOW()::DATE); | ||
| SELECT create_v1_range_partition('v1_dag_to_task_partitioned', (NOW() + INTERVAL '1 day')::DATE); |
There was a problem hiding this comment.
Shouldn't there be corresponding code changes to maintain this partition?
Also: would weekly partitions make sense here, or daily?
There was a problem hiding this comment.
yup, thanks 🤦 keep forgetting this when going from cloud to the oss
re: weekly vs daily - I guess weekly is fine? it'd be easier on the planner
mrkaye97
force-pushed
the
mk/partition-core-dag-to-task
branch
from
4d4e2ca to
8b80f3d
Compare
mrkaye97
force-pushed
the
mk/partition-core-dag-to-task
branch
from
fe24e64 to
605ddae
Compare
This reverts commit 5fca616.
mrkaye97
force-pushed
the
mk/partition-core-dag-to-task
branch
from
5c28693 to
92f771d
Compare
Description
Partitioning
v1_dag_to_task- this one was much simpler, since we can just create a partition using all of the existing data and attach it to the new tableType of change