feat: add automated migrations to deployment pipeline

[jakebromberg]

Summary

• Add database indexes for performance optimization on frequently-queried foreign key columns
• Automate database migrations as part of the deployment pipeline
• Fix orphaned migration file that was not tracked in Drizzle's journal
• NEW: Add comprehensive migration testing and analysis tools

Changes

Migration Testing Tools (New)

Added infrastructure to catch migration issues before production deployment:

Tool	Purpose
lint-migrations.js	Static analysis for dangerous patterns
generate-rollback.mjs	Auto-generate rollback SQL scripts
estimate-migration.mjs	Estimate migration duration based on table sizes
create-schema-snapshot.mjs	Export anonymized production schema
load-schema-snapshot.mjs	Load snapshot into test database
test-migrations-snapshot.mjs	Test migrations against production-like data

Lint Rules:

• concurrent-index: Warns when CREATE INDEX misses CONCURRENTLY
• not-null-no-default: Blocks ADD COLUMN NOT NULL without DEFAULT
• missing-if-not-exists: Warns about missing idempotency guards
• dangerous-ops: Blocks TRUNCATE, DROP TABLE without IF EXISTS

CI Integration:

• Migration linting runs on PRs that modify shared/
• Weekly snapshot testing workflow (Sundays 6 AM)

npm Scripts:

npm run lint:migrations          # Lint all migrations
npm run lint:migrations:changed  # Lint changed only (for CI)
npm run rollback:generate        # Generate rollback templates
npm run migration:estimate       # Estimate durations
npm run snapshot:test            # Test against snapshot

Database Indexes

Added B-tree indexes to improve query performance:

Table	Index	Columns
flowsheet	flowsheet_show_id_idx	show_id
flowsheet	flowsheet_album_id_idx	album_id
flowsheet	flowsheet_rotation_id_idx	rotation_id
show_djs	show_djs_show_id_dj_id_idx	(show_id, dj_id)
show_djs	show_djs_dj_id_idx	dj_id
bins	bins_dj_id_idx	dj_id
bins	bins_album_id_idx	album_id

Deployment Pipeline

Added migration jobs to the deployment workflow:

flowchart TB
    subgraph setup[Setup]
        A[Detect targets] --> B[Detect migration version]
    end
    
    subgraph build_phase[Build Phase]
        C[build-migrate] --> D[Build migrate:NNNN image]
        E[build] --> F[Build app images]
    end
    
    subgraph deploy_phase[Deploy Phase]
        G[migrate] --> H[Run migrations on EC2]
        H --> I[deploy]
        I --> J[Deploy app containers]
    end
    
    setup --> build_phase
    build_phase --> deploy_phase

Loading

Migration versioning: Images are tagged with the highest migration number (e.g., migrate:0028), providing traceability without manual version management.

Migration Fixes

• Renamed orphaned 0024_anonymous_devices.sql → 0028_anonymous_devices.sql
• Added missing journal entry and snapshot files
• Fixed 0027_add-performance-indexes.sql to contain only index definitions

New Files

File	Purpose
Dockerfile.migrate	Lightweight image for running Drizzle migrations
.github/actions/run-migrations/action.yml	Reusable action to run migrations on EC2
.github/workflows/migration-snapshot-test.yml	Weekly snapshot testing workflow
scripts/lint-migrations.js	Migration static analysis
scripts/generate-rollback.mjs	Rollback script generator
scripts/estimate-migration.mjs	Duration estimator
scripts/*-snapshot.mjs	Snapshot testing scripts
shared/database/src/migrations/rollbacks/	Rollback scripts for all migrations

Test plan

• CI passes (migrations run against fresh database in integration tests)
• Manual: verify Dockerfile.migrate builds locally
• Manual: run npm run lint:migrations to verify linter works
• Manual: run npm run migration:estimate 0027_add-performance-indexes.sql
• First deployment: verify migrations run before app deployment

Notes

Migrations are already tested in CI via init-db.mjs which runs npm run drizzle:migrate during test environment setup. Any PR that modifies shared/** (including migrations) triggers integration tests.