chore: use partman for archiving #459
Conversation
maschad
added
enhancement
There was a problem hiding this comment.
Pull Request Overview
This PR introduces database maintenance scripts utilizing pg_partman for archiving and partitioning along with configuration updates for S3 exports and monitoring. Key changes include new shell and SQL scripts for partition management, restoration from S3, and archiving operations, as well as updates to docker-compose, Ofelia, and Prometheus configuration for automated scheduling and monitoring.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/db-maintenance/run-maintenance.sh | New bash script to install extensions, initialize partitioning, and run archival processes. |
| scripts/db-maintenance/restore-from-s3.sql | SQL function to restore data from S3 with proper logging and error handling. |
| scripts/db-maintenance/init-partioning.sql | SQL script to set up partitioning using pg_partman (note a possible naming typo). |
| scripts/db-maintenance/archive-partitions.sql | SQL script that archives partitions to S3 and creates local backups. |
| scripts/db-maintenance/.env.example | AWS S3 configuration sample file for database archiving. |
| prometheus.yml | Added scrape configuration for monitoring the archival process. |
| ofelia.ini | Added cron job configuration for monthly archival maintenance. |
| docker-compose.yaml | Added postgres-maintenance and postgres-cron services for DB maintenance scheduling. |
| p_template_table = > NULL, | ||
| p_retention = > retention_months || ' months', | ||
| p_retention_keep_table = > true, | ||
| p_start_partition = > partition_start, | ||
| p_premake = > 3, | ||
| p_automatic_maintenance = > 'on' |
There was a problem hiding this comment.
The parameter assignment operator in the partman.create_parent call is incorrectly spaced; replace '= >' with '=>' to ensure proper SQL syntax.
| p_template_table = > NULL, | |
| p_retention = > retention_months || ' months', | |
| p_retention_keep_table = > true, | |
| p_start_partition = > partition_start, | |
| p_premake = > 3, | |
| p_automatic_maintenance = > 'on' | |
| p_template_table => NULL, | |
| p_retention => retention_months || ' months', | |
| p_retention_keep_table => true, | |
| p_start_partition => partition_start, | |
| p_premake => 3, | |
| p_automatic_maintenance => 'on' |
| psql -v ON_ERROR_STOP=1 -d "$POSTGRES_DB" -c "CREATE EXTENSION IF NOT EXISTS aws_s3;" | ||
|
|
||
| # Initialize the partitioning schema | ||
| psql -v ON_ERROR_STOP=1 -d "$POSTGRES_DB" -f /scripts/init-partitioning.sql |
There was a problem hiding this comment.
[nitpick] Verify that the file name '/scripts/init-partitioning.sql' matches the intended naming; if the filename is meant to be 'init-partitioning.sql', consider correcting 'init-partioning.sql' for consistency.
More info is explained in
partmanand s3 for archival partitions atoma-node#610We will need a back up of records as well as going forward once fiat payments are integrated we will also need to store user stack related info for the foreseeable future, which means it will need to be archived at intervals to avoid deletion.
Also we will need backups for any db maintenance or migrations
This integrates https://github.com/pgpartman/pg_partman to manage archiving via s3