Write migration guide from standard Coolify to enterprise

[johnproblems]

Task: Write migration guide from standard Coolify to enterprise

Description

Create a comprehensive migration guide that enables existing Coolify users to upgrade from the standard open-source version to the enterprise multi-tenant platform. This guide must provide clear, step-by-step instructions for database schema migration, data transformation, and system configuration while minimizing downtime and ensuring data integrity throughout the migration process.

The migration from standard Coolify to Enterprise is a complex operation involving:

1. Organizational Transformation: Converting the team-based structure to a hierarchical organization model (Top Branch → Master Branch → Sub-Users → End Users)
2. Schema Evolution: Adding enterprise tables (organizations, licenses, white-label configs, cloud credentials, Terraform deployments, resource monitoring)
3. Data Migration: Transforming existing teams into organizations, preserving all applications, servers, databases, and deployment configurations
4. Infrastructure Enhancement: Installing new dependencies (Terraform, SASS compiler, enhanced monitoring)
5. Service Deployment: Configuring new enterprise services (licensing, branding, capacity management)
6. Zero-Downtime Strategy: Implementing blue-green deployment with rollback capability

Target Audience:

• System administrators managing Coolify installations
• DevOps engineers responsible for platform operations
• Enterprise customers upgrading from open-source Coolify
• Technical decision-makers evaluating enterprise adoption

Critical Success Factors:

• Data Integrity: Zero data loss during migration
• Minimal Downtime: < 1 hour of service interruption for migrations
• Reversibility: Complete rollback capability if migration fails
• Validation: Comprehensive pre/post-migration verification
• Documentation: Clear troubleshooting steps for common issues

Why this task is important: The migration guide is the bridge between standard Coolify and the enterprise platform. Without comprehensive, tested documentation, customers risk data loss, extended downtime, and failed migrations. A well-crafted guide reduces migration risk, builds customer confidence, and accelerates enterprise adoption. This document becomes the single source of truth for migration operations and serves as the foundation for customer success during the critical upgrade phase.

Acceptance Criteria

• Complete pre-migration checklist with system requirements verification
• Detailed database backup and restore procedures
• Step-by-step migration instructions with exact commands
• Data transformation scripts for team → organization conversion
• Zero-downtime migration strategy documentation
• Rollback procedures for each migration phase
• Post-migration verification checklist
• Common issues troubleshooting section with resolutions
• Performance optimization recommendations post-migration
• Security hardening checklist for enterprise features
• Configuration examples for all new enterprise services
• Estimated timeline for each migration phase
• Downtime windows and maintenance mode procedures
• Multi-server migration considerations (distributed deployments)
• Docker container migration for existing applications

Technical Details

Documentation Structure

File Location: docs/migration/standard-to-enterprise.md

Supporting Files:

• docs/migration/scripts/pre-migration-check.sh - Pre-flight validation
• docs/migration/scripts/migrate-teams-to-orgs.php - Data transformation
• docs/migration/scripts/post-migration-verify.sh - Verification tests
• docs/migration/rollback/ - Rollback procedures for each phase
• docs/migration/examples/ - Configuration file examples

Documentation Outline

# Migration Guide: Standard Coolify to Enterprise Edition

## Table of Contents
1. Overview
2. Prerequisites and Requirements
3. Pre-Migration Planning
4. Backup and Safety Procedures
5. Migration Phases
6. Post-Migration Configuration
7. Verification and Testing
8. Rollback Procedures
9. Troubleshooting
10. FAQ

## 1. Overview

### What's New in Enterprise Edition
- Multi-tenant organization hierarchy
- Enterprise licensing system
- White-label branding
- Terraform infrastructure provisioning
- Advanced resource monitoring
- Payment processing integration
- Enhanced API with rate limiting

### Migration Scope
- Database schema evolution (40+ new tables)
- Team → Organization transformation
- New service deployment
- Infrastructure dependencies
- Configuration updates

### Estimated Timeline
- Small deployment (< 100 applications): 2-4 hours
- Medium deployment (100-1000 applications): 4-8 hours
- Large deployment (> 1000 applications): 8-16 hours

### Downtime Requirements
- Maintenance mode: 30-60 minutes
- Database migration: 15-30 minutes
- Verification: 15-30 minutes
- Total estimated downtime: 1-2 hours

## 2. Prerequisites and Requirements

### System Requirements

**Hardware Requirements:**
- CPU: 4+ cores (8+ recommended for large deployments)
- RAM: 8GB minimum, 16GB+ recommended
- Disk: 100GB+ free space (for database backups and logs)
- Network: 100Mbps+ internet connection

**Software Requirements:**
- PostgreSQL 15+
- PHP 8.4+
- Redis 7+
- Docker 24+
- Terraform 1.5+ (new dependency)
- Node.js 20+ (for asset compilation)
- Git (for version management)

### Pre-Migration Checklist

```bash
# Run pre-migration verification script
bash docs/migration/scripts/pre-migration-check.sh

# Expected checks:
✓ PostgreSQL version >= 15
✓ PHP version >= 8.4
✓ Redis version >= 7
✓ Docker version >= 24
✓ Sufficient disk space (100GB+)
✓ Database backup capabilities
✓ Network connectivity
✓ Terraform installation
✓ Current Coolify version >= 4.0

Backup Requirements

Full System Backup:

# 1. Database backup
pg_dump -U coolify coolify > coolify_backup_$(date +%Y%m%d_%H%M%S).sql

# 2. Application data backup
tar -czf coolify_data_$(date +%Y%m%d_%H%M%S).tar.gz /var/lib/docker/volumes

# 3. Configuration backup
tar -czf coolify_config_$(date +%Y%m%d_%H%M%S).tar.gz /data/coolify

# 4. Verify backups
ls -lh coolify_*
md5sum coolify_* > backup_checksums.txt

Dependencies Installation

# Install Terraform
wget https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_linux_amd64.zip
unzip terraform_1.5.7_linux_amd64.zip
sudo mv terraform /usr/local/bin/
terraform --version

# Install PHP dependencies
sudo apt-get update
sudo apt-get install -y php8.4-cli php8.4-fpm php8.4-pgsql php8.4-redis \
    php8.4-curl php8.4-xml php8.4-zip php8.4-mbstring php8.4-gd

# Install Node.js for asset compilation
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs

# Verify installations
php --version
node --version
npm --version
terraform --version

3. Pre-Migration Planning

Team → Organization Mapping Strategy

Option 1: One Team = One Organization (Recommended)

• Each existing team becomes a Top Branch Organization
• Team owner becomes Organization Administrator
• Team members become organization users with preserved roles

Option 2: Multiple Teams = Single Organization

• Consolidate related teams into a single organization
• Requires manual mapping configuration
• Useful for simplifying organizational structure

Option 3: Custom Mapping

• Define custom team-to-organization relationships
• Use migration configuration file
• Suitable for complex organizational structures

Migration Configuration File

Create config/migration.php:

<?php

return [
    'migration_strategy' => 'one_to_one', // one_to_one, consolidate, custom

    // Team to Organization mapping (for custom strategy)
    'team_mapping' => [
        // team_id => organization_name
        1 => 'Engineering Organization',
        2 => 'Engineering Organization', // Consolidate teams 1 & 2
        3 => 'Marketing Organization',
    ],

    // Default license for migrated organizations
    'default_license' => [
        'tier' => 'professional',
        'max_projects' => 50,
        'max_servers' => 10,
        'max_users' => 25,
        'features' => [
            'white_label' => true,
            'terraform_provisioning' => true,
            'advanced_monitoring' => true,
            'api_access' => true,
        ],
    ],

    // Migration options
    'options' => [
        'preserve_team_ids' => true, // Keep team IDs as organization IDs
        'migrate_permissions' => true,
        'migrate_api_tokens' => true,
        'migrate_webhooks' => true,
        'create_default_branding' => true,
    ],

    // Validation rules
    'validation' => [
        'verify_data_integrity' => true,
        'test_api_endpoints' => true,
        'check_application_accessibility' => true,
    ],
];

Maintenance Mode Planning

Communication Template:

Subject: Scheduled Maintenance - Coolify Enterprise Upgrade

Dear Coolify Users,

We will be upgrading to Coolify Enterprise Edition on [DATE] from [START_TIME] to [END_TIME].

Expected downtime: 1-2 hours

What to expect:
- Coolify UI will be unavailable during maintenance
- Existing applications will continue running (no application downtime)
- New deployments will be queued and processed after upgrade
- You will receive an email when the upgrade is complete

What's new in Enterprise Edition:
- Organization-based access control
- White-label branding capabilities
- Infrastructure provisioning via Terraform
- Advanced resource monitoring
- Enhanced API with rate limiting

We have completed extensive testing and have full backup procedures in place.

If you have any concerns, please contact: [SUPPORT_EMAIL]

Thank you for your patience.

4. Backup and Safety Procedures

Complete Backup Script

File: docs/migration/scripts/backup-all.sh

#!/bin/bash

set -e # Exit on error

BACKUP_DIR="/backup/coolify-migration-$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

echo "Starting full Coolify backup..."
echo "Backup directory: $BACKUP_DIR"

# 1. Database backup with compression
echo "Backing up database..."
pg_dump -U coolify -F c -b -v -f "$BACKUP_DIR/database.dump" coolify
echo "✓ Database backup completed"

# 2. Docker volumes backup
echo "Backing up Docker volumes..."
docker run --rm -v coolify-data:/data -v "$BACKUP_DIR":/backup \
    alpine tar czf /backup/docker-volumes.tar.gz /data
echo "✓ Docker volumes backup completed"

# 3. Configuration files backup
echo "Backing up configuration..."
tar czf "$BACKUP_DIR/config.tar.gz" \
    /data/coolify/source/.env \
    /data/coolify/source/config/ \
    /data/coolify/proxy/ \
    /data/coolify/ssh/
echo "✓ Configuration backup completed"

# 4. Application data backup
echo "Backing up application data..."
tar czf "$BACKUP_DIR/applications.tar.gz" \
    /data/coolify/applications/ \
    /data/coolify/databases/
echo "✓ Application data backup completed"

# 5. Generate checksums
echo "Generating checksums..."
cd "$BACKUP_DIR"
sha256sum * > checksums.sha256
echo "✓ Checksums generated"

# 6. Backup verification
echo "Verifying backups..."
sha256sum -c checksums.sha256
echo "✓ Backup verification completed"

# 7. Create backup manifest
cat > "$BACKUP_DIR/manifest.json" <<EOF
{
    "backup_timestamp": "$(date -Iseconds)",
    "coolify_version": "$(cd /data/coolify/source && git describe --tags)",
    "database_size_mb": $(stat -f%z "$BACKUP_DIR/database.dump" 2>/dev/null || stat -c%s "$BACKUP_DIR/database.dump" | awk '{print $1/1024/1024}'),
    "backup_directory": "$BACKUP_DIR",
    "checksum_file": "checksums.sha256"
}
EOF

echo ""
echo "=========================================="
echo "Backup completed successfully!"
echo "=========================================="
echo "Backup location: $BACKUP_DIR"
echo "Backup size: $(du -sh $BACKUP_DIR | cut -f1)"
echo ""
echo "To restore from this backup, run:"
echo "  bash docs/migration/scripts/restore-backup.sh $BACKUP_DIR"
echo ""

Backup Verification Checklist

• Database backup file exists and is non-empty
• Database backup can be listed: pg_restore -l database.dump
• Docker volumes backup size matches expected size
• Configuration files backup includes .env file
• All checksums verify correctly
• Backup manifest is readable and contains correct information
• Total backup size is reasonable (estimate: 2-10GB typical)
• Backup stored on separate disk/server (not same as Coolify)

Restore Test Procedure

Test restoration before migration:

# Create test restoration environment
docker run -d --name postgres-test -e POSTGRES_PASSWORD=test postgres:15

# Restore database backup to test instance
pg_restore -U postgres -d postgres -C test -v database.dump

# Verify restoration
psql -U postgres -d coolify_test -c "SELECT COUNT(*) FROM teams;"
psql -U postgres -d coolify_test -c "SELECT COUNT(*) FROM applications;"

# Cleanup test environment
docker stop postgres-test
docker rm postgres-test

5. Migration Phases

Phase 1: Enable Maintenance Mode (5 minutes)

# 1. Enable maintenance mode
cd /data/coolify/source
php artisan down --render="errors::503" --secret="migration-$(openssl rand -hex 16)"

# Save the secret token for admin access
echo "Admin bypass URL: https://your-coolify.com/migration-{SECRET}"

# 2. Verify maintenance mode
curl -I https://your-coolify.com
# Should return HTTP 503

# 3. Drain pending jobs
php artisan queue:restart
php artisan horizon:pause

# Wait for running jobs to complete (check Horizon dashboard)
# Timeout: 5 minutes maximum

Phase 2: Pull Enterprise Code (10 minutes)

# 1. Add enterprise repository remote
cd /data/coolify/source
git remote add enterprise https://github.com/your-org/coolify-enterprise.git

# 2. Fetch enterprise branch
git fetch enterprise v4.x-enterprise

# 3. Create backup branch
git branch backup-pre-enterprise

# 4. Checkout enterprise branch
git checkout -b enterprise-migration enterprise/v4.x-enterprise

# 5. Install new dependencies
composer install --no-dev --optimize-autoloader
npm ci
npm run build

# 6. Verify installation
php artisan --version
# Should show: "Laravel Framework 12.x.x"

Phase 3: Database Migration (15-30 minutes)

# 1. Review pending migrations
php artisan migrate:status

# Expected new migrations:
# - create_organizations_table
# - create_organization_users_table
# - create_enterprise_licenses_table
# - create_white_label_configs_table
# - create_cloud_provider_credentials_table
# - create_terraform_deployments_table
# - create_server_resource_metrics_table
# - create_organization_resource_usage_table
# ... (30+ new tables)

# 2. Run migrations with backup
php artisan migrate --force 2>&1 | tee migration_log.txt

# 3. Verify migration success
php artisan migrate:status | grep "Ran"

# 4. Check for migration errors
grep -i error migration_log.txt
# Should return no results

Phase 4: Data Transformation (20-60 minutes)

Run the team-to-organization conversion script:

File: docs/migration/scripts/migrate-teams-to-orgs.php

<?php

use App\Models\Team;
use App\Models\Organization;
use App\Models\User;
use App\Models\Server;
use App\Models\Application;
use App\Models\Database;
use App\Models\EnterpriseLicense;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;

/**
 * Migrate teams to organizations
 *
 * This script transforms the team-based structure to organization hierarchy:
 * - Each team becomes a Top Branch Organization
 * - Team members become organization users with preserved roles
 * - All resources (servers, apps, databases) are transferred
 * - Default enterprise licenses are created
 */

require __DIR__.'/../../vendor/autoload.php';

$app = require_once __DIR__.'/../../bootstrap/app.php';
$app->make(Illuminate\Contracts\Console\Kernel::class)->bootstrap();

$migrationConfig = config('migration');

echo "============================================\n";
echo "Team to Organization Migration\n";
echo "============================================\n\n";

DB::transaction(function () use ($migrationConfig) {
    $teams = Team::with(['users', 'servers', 'applications', 'databases'])->get();

    $progressBar = new \Symfony\Component\Console\Helper\ProgressBar(
        new \Symfony\Component\Console\Output\ConsoleOutput(),
        $teams->count()
    );

    $progressBar->start();

    foreach ($teams as $team) {
        try {
            // 1. Create organization from team
            $organization = Organization::create([
                'id' => $migrationConfig['options']['preserve_team_ids'] ? $team->id : null,
                'name' => $team->name,
                'slug' => \Illuminate\Support\Str::slug($team->name),
                'type' => 'top_branch', // Top-level organization
                'parent_organization_id' => null,
                'description' => "Migrated from team: {$team->name}",
                'created_at' => $team->created_at,
            ]);

            // 2. Create enterprise license
            EnterpriseLicense::create([
                'organization_id' => $organization->id,
                'license_key' => 'MIGRATED-' . strtoupper(bin2hex(random_bytes(16))),
                'tier' => $migrationConfig['default_license']['tier'],
                'status' => 'active',
                'max_projects' => $migrationConfig['default_license']['max_projects'],
                'max_servers' => $migrationConfig['default_license']['max_servers'],
                'max_users' => $migrationConfig['default_license']['max_users'],
                'features' => $migrationConfig['default_license']['features'],
                'valid_from' => now(),
                'valid_until' => now()->addYear(),
            ]);

            // 3. Migrate team members to organization users
            foreach ($team->users as $user) {
                $pivot = $user->pivot; // TeamUser relationship

                $organization->users()->attach($user->id, [
                    'role' => $pivot->role, // Preserve role (owner, admin, member)
                    'permissions' => $pivot->permissions ?? [],
                    'joined_at' => $pivot->created_at,
                ]);
            }

            // 4. Transfer servers to organization
            Server::where('team_id', $team->id)->update([
                'organization_id' => $organization->id,
            ]);

            // 5. Transfer applications to organization
            Application::where('team_id', $team->id)->update([
                'organization_id' => $organization->id,
            ]);

            // 6. Transfer databases to organization
            Database::where('team_id', $team->id)->update([
                'organization_id' => $organization->id,
            ]);

            // 7. Create default white-label config
            if ($migrationConfig['options']['create_default_branding']) {
                $organization->whiteLabelConfig()->create([
                    'platform_name' => $organization->name . ' Platform',
                    'primary_color' => '#3b82f6',
                    'secondary_color' => '#8b5cf6',
                    'accent_color' => '#10b981',
                    'font_family' => 'Inter, sans-serif',
                ]);
            }

            Log::info("Migrated team to organization", [
                'team_id' => $team->id,
                'organization_id' => $organization->id,
                'users_count' => $team->users->count(),
                'servers_count' => $team->servers->count(),
                'applications_count' => $team->applications->count(),
            ]);

            $progressBar->advance();

        } catch (\Exception $e) {
            Log::error("Failed to migrate team", [
                'team_id' => $team->id,
                'error' => $e->getMessage(),
                'trace' => $e->getTraceAsString(),
            ]);

            throw $e; // Rollback transaction
        }
    }

    $progressBar->finish();
    echo "\n\n";
});

echo "============================================\n";
echo "Migration completed successfully!\n";
echo "============================================\n";

// Display summary
$organizationCount = Organization::count();
$userCount = DB::table('organization_users')->count();
$serverCount = Server::whereNotNull('organization_id')->count();
$applicationCount = Application::whereNotNull('organization_id')->count();

echo "Summary:\n";
echo "- Organizations created: {$organizationCount}\n";
echo "- Organization users: {$userCount}\n";
echo "- Servers migrated: {$serverCount}\n";
echo "- Applications migrated: {$applicationCount}\n";
echo "\n";

Run the migration:

php docs/migration/scripts/migrate-teams-to-orgs.php

Phase 5: Service Configuration (15 minutes)

# 1. Update environment variables
cat >> /data/coolify/source/.env <<EOF

# Enterprise Configuration
ENTERPRISE_MODE=true
TERRAFORM_BINARY_PATH=/usr/local/bin/terraform
WHITE_LABEL_CACHE_TTL=3600
ORGANIZATION_DEFAULT_QUOTAS='{"max_projects":50,"max_servers":10,"max_users":25}'

# Licensing (configure with your license server)
LICENSE_SERVER_URL=https://license.your-company.com
LICENSE_ENCRYPTION_KEY=$(openssl rand -base64 32)

# Payment Processing (optional, configure later)
# PAYMENT_STRIPE_SECRET_KEY=sk_live_...
# PAYMENT_PAYPAL_CLIENT_ID=...

EOF

# 2. Clear caches
php artisan cache:clear
php artisan config:clear
php artisan route:clear
php artisan view:clear

# 3. Rebuild caches
php artisan config:cache
php artisan route:cache
php artisan view:cache

# 4. Run seeders for enterprise data
php artisan db:seed --class=EnterpriseSeeder

# 5. Restart services
php artisan queue:restart
php artisan horizon:terminate # Will auto-restart via supervisor
php artisan reverb:restart

Phase 6: Asset Compilation (10 minutes)

# 1. Compile Vue.js enterprise components
npm run build

# 2. Publish assets
php artisan vendor:publish --tag=enterprise-assets

# 3. Optimize assets
php artisan optimize

# 4. Verify assets
ls -la public/build/
# Should show compiled manifest.json and asset files

Phase 7: Disable Maintenance Mode (5 minutes)

# 1. Run final verification
php docs/migration/scripts/post-migration-verify.sh

# 2. Disable maintenance mode
php artisan up

# 3. Verify site is accessible
curl -I https://your-coolify.com
# Should return HTTP 200

# 4. Restart queue workers
php artisan horizon:continue

6. Post-Migration Configuration

Organization Configuration

# 1. List migrated organizations
php artisan tinker
>>> Organization::all()->pluck('name', 'id');

# 2. Update organization details (if needed)
>>> $org = Organization::find(1);
>>> $org->update(['description' => 'Engineering team organization']);

# 3. Verify license assignment
>>> $org->license()->exists(); // Should return true

User Access Verification

# Verify users can access their organizations
php artisan tinker
>>> User::with('organizations')->find(1)->organizations;

# Test user login and organization switching
# Manually test in browser:
# 1. Login as existing user
# 2. Should see organization switcher in UI
# 3. Verify access to existing applications

API Token Migration

# Re-issue API tokens with organization context
php artisan enterprise:migrate-api-tokens

# This script:
# - Updates existing tokens with organization_id
# - Adds organization scope to token abilities
# - Notifies users to regenerate tokens (optional)

Webhook Configuration

# Update webhooks with new organization context
php artisan enterprise:update-webhooks

# Verify webhooks are working
curl -X POST https://your-coolify.com/webhooks/test \
    -H "Authorization: Bearer {token}"

White-Label Branding Setup

For each organization that wants custom branding:

# 1. Access Branding Settings
# Navigate to: https://your-coolify.com/enterprise/organizations/{id}/branding

# 2. Upload logo (via UI or API)
curl -X POST https://your-coolify.com/api/v1/organizations/{id}/branding/logo \
    -H "Authorization: Bearer {token}" \
    -F "logo=@company-logo.png" \
    -F "logo_type=primary"

# 3. Configure colors
curl -X PUT https://your-coolify.com/api/v1/organizations/{id}/branding \
    -H "Authorization: Bearer {token}" \
    -H "Content-Type: application/json" \
    -d '{
        "primary_color": "#FF5733",
        "secondary_color": "#3366FF",
        "platform_name": "Custom Platform Name"
    }'

# 4. Generate favicons
php artisan branding:generate-favicons {organization_id}

7. Verification and Testing

Post-Migration Verification Script

File: docs/migration/scripts/post-migration-verify.sh

#!/bin/bash

set -e

echo "============================================"
echo "Post-Migration Verification"
echo "============================================"
echo ""

ERRORS=0

# Test 1: Database connectivity
echo "Testing database connectivity..."
if php artisan tinker --execute="DB::connection()->getPdo();" &>/dev/null; then
    echo "✓ Database connection successful"
else
    echo "✗ Database connection failed"
    ((ERRORS++))
fi

# Test 2: Organizations exist
echo "Testing organizations..."
ORG_COUNT=$(php artisan tinker --execute="echo Organization::count();")
if [ "$ORG_COUNT" -gt 0 ]; then
    echo "✓ Organizations exist ($ORG_COUNT found)"
else
    echo "✗ No organizations found"
    ((ERRORS++))
fi

# Test 3: Licenses exist
echo "Testing licenses..."
LICENSE_COUNT=$(php artisan tinker --execute="echo EnterpriseLicense::count();")
if [ "$LICENSE_COUNT" -gt 0 ]; then
    echo "✓ Licenses exist ($LICENSE_COUNT found)"
else
    echo "✗ No licenses found"
    ((ERRORS++))
fi

# Test 4: Applications accessible
echo "Testing application accessibility..."
APP_COUNT=$(php artisan tinker --execute="echo Application::whereNotNull('organization_id')->count();")
if [ "$APP_COUNT" -gt 0 ]; then
    echo "✓ Applications migrated ($APP_COUNT found)"
else
    echo "✗ No applications found with organization_id"
    ((ERRORS++))
fi

# Test 5: Servers accessible
echo "Testing server accessibility..."
SERVER_COUNT=$(php artisan tinker --execute="echo Server::whereNotNull('organization_id')->count();")
if [ "$SERVER_COUNT" -gt 0 ]; then
    echo "✓ Servers migrated ($SERVER_COUNT found)"
else
    echo "✗ No servers found with organization_id"
    ((ERRORS++))
fi

# Test 6: API endpoint health
echo "Testing API endpoints..."
if curl -sf https://your-coolify.com/api/health &>/dev/null; then
    echo "✓ API health endpoint responding"
else
    echo "✗ API health endpoint not responding"
    ((ERRORS++))
fi

# Test 7: Vue.js assets compiled
echo "Testing Vue.js assets..."
if [ -f public/build/manifest.json ]; then
    echo "✓ Vue.js assets compiled"
else
    echo "✗ Vue.js assets not found"
    ((ERRORS++))
fi

# Test 8: Terraform binary available
echo "Testing Terraform installation..."
if command -v terraform &>/dev/null; then
    TERRAFORM_VERSION=$(terraform --version | head -n1)
    echo "✓ Terraform installed ($TERRAFORM_VERSION)"
else
    echo "✗ Terraform not found"
    ((ERRORS++))
fi

# Test 9: Queue workers running
echo "Testing queue workers..."
if php artisan horizon:status | grep -q "running"; then
    echo "✓ Horizon workers running"
else
    echo "✗ Horizon workers not running"
    ((ERRORS++))
fi

# Test 10: Cache working
echo "Testing cache..."
if php artisan tinker --execute="Cache::put('test', 'value', 60); echo Cache::get('test');" | grep -q "value"; then
    echo "✓ Cache working"
else
    echo "✗ Cache not working"
    ((ERRORS++))
fi

echo ""
echo "============================================"
if [ $ERRORS -eq 0 ]; then
    echo "✓ All verification tests passed!"
    echo "============================================"
    exit 0
else
    echo "✗ $ERRORS verification test(s) failed"
    echo "============================================"
    echo "Please review errors above before proceeding."
    exit 1
fi

Manual Verification Checklist

• Login with existing user credentials
• Organization switcher appears in UI
• Access existing applications from organization dashboard
• Deploy a test application successfully
• View server metrics and logs
• Create new organization (if admin)
• Upload branding logo (test white-label)
• Generate and verify custom CSS is applied
• Test API with existing tokens
• Verify webhooks trigger correctly
• Check background jobs are processing (Horizon dashboard)
• Verify email notifications are sent

Performance Testing

# 1. Test dashboard load time
time curl -o /dev/null -s -w "Time: %{time_total}s\n" \
    https://your-coolify.com/dashboard

# Should be < 2 seconds

# 2. Test API response time
for i in {1..10}; do
    time curl -o /dev/null -s -w "Time: %{time_total}s\n" \
        -H "Authorization: Bearer {token}" \
        https://your-coolify.com/api/v1/organizations
done | awk '{sum+=$2; count++} END {print "Average:", sum/count, "seconds"}'

# Should be < 500ms

# 3. Test database query performance
php artisan tinker --execute="
    \$start = microtime(true);
    Organization::with(['users', 'servers', 'applications'])->get();
    echo 'Query time: ' . round((microtime(true) - \$start) * 1000) . 'ms';
"

# Should be < 100ms for small datasets

8. Rollback Procedures

When to Rollback

Initiate rollback if:

• Data integrity issues detected (missing resources, incorrect counts)
• Critical features not working (unable to deploy, servers unreachable)
• Performance degradation (> 5x slower than pre-migration)
• Errors in verification tests (> 2 failures)

Rollback Procedure (30 minutes)

# 1. Enable maintenance mode
cd /data/coolify/source
php artisan down

# 2. Stop all services
php artisan horizon:terminate
php artisan queue:restart
systemctl stop coolify-reverb

# 3. Restore database
pg_restore -U coolify -d coolify --clean --if-exists \
    /backup/coolify-migration-{TIMESTAMP}/database.dump

# 4. Restore configuration
tar xzf /backup/coolify-migration-{TIMESTAMP}/config.tar.gz -C /

# 5. Restore Docker volumes
docker run --rm -v coolify-data:/data -v /backup/coolify-migration-{TIMESTAMP}:/backup \
    alpine sh -c "cd /data && tar xzf /backup/docker-volumes.tar.gz --strip 1"

# 6. Checkout previous version
git checkout backup-pre-enterprise

# 7. Install previous dependencies
composer install --no-dev
npm ci
npm run build

# 8. Clear caches
php artisan cache:clear
php artisan config:clear

# 9. Restart services
systemctl start coolify-reverb
php artisan horizon:start

# 10. Disable maintenance mode
php artisan up

# 11. Verify restoration
bash docs/migration/scripts/verify-rollback.sh

Post-Rollback Verification

# Verify counts match pre-migration snapshot
echo "Teams: $(php artisan tinker --execute='echo Team::count();')"
echo "Users: $(php artisan tinker --execute='echo User::count();')"
echo "Servers: $(php artisan tinker --execute='echo Server::count();')"
echo "Applications: $(php artisan tinker --execute='echo Application::count();')"

# Compare with pre-migration snapshot:
cat /backup/coolify-migration-{TIMESTAMP}/pre-migration-snapshot.txt

Rollback Communication Template

Subject: Coolify Enterprise Migration - Rollback Completed

Dear Coolify Users,

We encountered issues during the enterprise migration and have rolled back to the previous version.

Current status: All services restored and operational

Details:
- Database restored from backup (snapshot: {TIMESTAMP})
- All applications and servers are accessible
- Previous functionality fully restored
- No data loss occurred

Next steps:
- We are analyzing the migration issues
- A new migration date will be scheduled after fixes
- You will receive advance notice of the new migration window

We apologize for any inconvenience caused.

If you experience any issues, please contact: [SUPPORT_EMAIL]

9. Troubleshooting

Common Issues and Resolutions

Issue 1: Migration Timeout

Symptom: Migration hangs or times out during database migration phase

Resolution:

# Increase PHP timeout
echo "max_execution_time = 3600" >> /etc/php/8.4/cli/php.ini

# Increase PostgreSQL statement timeout
psql -U coolify -c "ALTER DATABASE coolify SET statement_timeout = '3600s';"

# Re-run failed migration
php artisan migrate --force

Issue 2: Team-to-Organization Script Fails

Symptom: migrate-teams-to-orgs.php script throws exceptions

Resolution:

# Check for duplicate team names
php artisan tinker --execute="
    Team::select('name')
        ->groupBy('name')
        ->havingRaw('COUNT(*) > 1')
        ->get();
"

# Manually rename duplicates before migration
php artisan tinker --execute="
    Team::where('name', 'Engineering')->skip(1)->first()->update(['name' => 'Engineering 2']);
"

# Re-run migration script
php docs/migration/scripts/migrate-teams-to-orgs.php

Issue 3: Missing Organization IDs

Symptom: Applications or servers still have team_id but no organization_id

Resolution:

# Fix orphaned resources
php artisan enterprise:fix-orphaned-resources

# Or manual fix:
php artisan tinker --execute="
    Application::whereNotNull('team_id')
        ->whereNull('organization_id')
        ->each(function(\$app) {
            \$org = Organization::where('id', \$app->team_id)->first();
            if (\$org) {
                \$app->update(['organization_id' => \$org->id]);
            }
        });
"

Issue 4: White-Label CSS Not Loading

Symptom: Custom branding CSS returns 404 or shows default styles

Resolution:

# Clear branding cache
php artisan cache:forget 'branding:*'

# Regenerate CSS for all organizations
php artisan branding:warmup-cache

# Verify CSS route is registered
php artisan route:list | grep branding

# Check file permissions
chmod -R 755 storage/app/public/branding
chown -R www-data:www-data storage/app/public/branding

Issue 5: API Tokens Invalid After Migration

Symptom: API requests return 401 Unauthorized after migration

Resolution:

# Re-migrate API tokens with organization context
php artisan enterprise:migrate-api-tokens --force

# Or instruct users to regenerate tokens:
# 1. Login to Coolify
# 2. Navigate to Settings → API Tokens
# 3. Delete old token
# 4. Create new token with organization scope

Issue 6: Terraform Commands Fail

Symptom: Infrastructure provisioning fails with "terraform: command not found"

Resolution:

# Verify Terraform installation
which terraform
# If not found, install:

wget https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_linux_amd64.zip
unzip terraform_1.5.7_linux_amd64.zip
sudo mv terraform /usr/local/bin/
terraform --version

# Update Terraform path in .env
echo "TERRAFORM_BINARY_PATH=/usr/local/bin/terraform" >> .env
php artisan config:clear
php artisan config:cache

Issue 7: High Memory Usage After Migration

Symptom: Server memory usage increases significantly post-migration

Resolution:

# Optimize database indexes
php artisan db:optimize

# Enable query caching for organization scopes
php artisan cache:config

# Reduce Horizon workers if needed
# Edit config/horizon.php:
'environments' => [
    'production' => [
        'supervisor-1' => [
            'processes' => 3, // Reduce from 10 to 3
        ],
    ],
],

# Restart Horizon
php artisan horizon:terminate

Issue 8: WebSocket Connection Failures

Symptom: Real-time updates not working in Vue.js components

Resolution:

# Check Reverb is running
php artisan reverb:status

# Restart Reverb
php artisan reverb:restart

# Verify WebSocket configuration
cat .env | grep REVERB

# Expected:
# REVERB_APP_ID=...
# REVERB_APP_KEY=...
# REVERB_APP_SECRET=...
# REVERB_HOST=0.0.0.0
# REVERB_PORT=8080
# REVERB_SCHEME=http

# Test WebSocket connection
wscat -c ws://localhost:8080/app/{REVERB_APP_KEY}

Performance Optimization Post-Migration

# 1. Optimize database queries
php artisan model:prune

# 2. Enable OPcache
echo "opcache.enable=1" >> /etc/php/8.4/fpm/php.ini
echo "opcache.memory_consumption=256" >> /etc/php/8.4/fpm/php.ini
systemctl restart php8.4-fpm

# 3. Enable Redis for sessions and cache
php artisan cache:clear
php artisan config:cache
# Sessions will automatically use Redis

# 4. Optimize Composer autoloader
composer dump-autoload --optimize --classmap-authoritative

# 5. Queue optimization
# Edit config/queue.php to use Redis instead of database
'default' => env('QUEUE_CONNECTION', 'redis'),

10. FAQ

Q: Will my existing applications experience downtime during migration?

A: No, running applications continue to operate during migration. Only the Coolify control panel will be unavailable during the 1-2 hour maintenance window.

Q: Do I need to update DNS records or change application URLs?

A: No, all application URLs, domains, and DNS records remain unchanged.

Q: Will API tokens continue to work after migration?

A: Existing tokens will work but should be regenerated to include organization context for proper scoping. Run php artisan enterprise:migrate-api-tokens to update existing tokens automatically.

Q: Can I migrate a subset of teams first?

A: The migration script migrates all teams in a single operation. Partial migrations are not supported.

Q: What happens to existing team permissions?

A: All team permissions are preserved and mapped to equivalent organization roles:

• Team Owner → Organization Administrator
• Team Admin → Organization Administrator
• Team Member → Organization Member

Q: How long should I keep the backup after migration?

A: Keep backups for at least 30 days. After successful operation for 30 days, you can archive or delete backups.

Q: Can I rollback after a few days of using Enterprise edition?

A: Rollback is only supported immediately after migration (same day). Once you've been using Enterprise features for several days, rollback becomes data-destructive.

Q: Is the migration reversible if I want to go back to open-source Coolify?

A: Technically yes, but you'll lose all enterprise features (organizations, white-label branding, advanced monitoring). A downgrade script is not provided but can be created on request.

Q: Do I need to notify users before migration?

A: Yes, send notification at least 48 hours in advance with maintenance window details.

Q: What if I find issues after disabling maintenance mode?

A: You have a 4-hour window for immediate rollback. After 4 hours, contact support for assistance with data fixes.

Q: Will webhook URLs change after migration?

A: Webhook URLs remain the same, but webhooks will now include organization context in payloads.

Q: Do I need to update Docker images for existing applications?

A: No, existing Docker images and containers are not affected by the migration.

Q: How do I migrate if I have multiple Coolify instances?

A: Migrate each instance separately. There's no shared state between instances.

Q: Can I test the migration on a staging environment first?

A: Highly recommended! Clone your production database to staging and run the full migration process there first.

Q: What are the licensing costs for Enterprise edition?

A: Contact sales for enterprise licensing. Pricing is based on number of organizations and resources.

11. Support and Resources

Getting Help

Pre-Migration Support:

• Email: enterprise-migration@your-company.com
• Slack: #coolify-enterprise-migration
• Schedule consultation: https://your-company.com/book-migration-call

Post-Migration Support:

• Create support ticket: https://your-company.com/support
• Emergency hotline: +1-XXX-XXX-XXXX (24/7 during migration window)

Additional Documentation

• Enterprise Features Overview
• Organization Management Guide
• White-Label Branding Guide
• Terraform Integration Guide
• API Documentation

Migration Checklist

Download the complete migration checklist: migration-checklist.pdf

Video Tutorials

• Migration Overview (15 minutes): [Watch Video]
• Step-by-Step Walkthrough (45 minutes): [Watch Video]
• Troubleshooting Common Issues (20 minutes): [Watch Video]