Enhance CI/CD pipeline with multi-environment deployment


# Task: Enhance CI/CD pipeline with multi-environment deployment

## Description

Create a comprehensive multi-environment CI/CD pipeline for the Coolify Enterprise transformation that supports automated deployment to development, staging, and production environments. This task establishes the foundation for reliable, repeatable deployments with environment-specific configurations, automated testing at each stage, database migration validation, zero-downtime deployments, and rollback capabilities.

**The Problem:**

The current Coolify project lacks a structured multi-environment deployment pipeline for the enterprise features being developed. Without this infrastructure, teams face:

1. **Manual deployment processes** prone to human error and configuration drift
2. **Inconsistent environments** leading to "works on my machine" issues
3. **No automated testing gates** before production deployments
4. **Lack of rollback capability** when deployments fail
5. **Database migration risks** without validation mechanisms
6. **Configuration management chaos** with environment-specific settings scattered across files

**The Solution:**

A modern CI/CD pipeline leveraging GitHub Actions (or existing CI/CD infrastructure) with:

- **Environment Separation**: Dev, staging, and production environments with distinct configurations
- **Automated Quality Gates**: Tests, static analysis, and security scans at each stage
- **Progressive Deployment**: Dev → Staging → Production with manual approval for production
- **Database Migration Safety**: Validation, backup, and rollback capabilities
- **Zero-Downtime Deployments**: Blue-green or rolling deployment strategies
- **Environment Parity**: Consistent infrastructure across environments using Docker/Terraform
- **Secret Management**: Secure handling of API keys, database credentials, cloud credentials
- **Monitoring Integration**: Automatic health checks and alerting after deployments

**Why This Task is Critical:**

The Coolify Enterprise transformation adds significant complexity (multi-tenancy, licensing, Terraform integration, payment processing). Without a robust CI/CD pipeline:

- **Production outages** become more likely as code complexity increases
- **Manual testing** becomes impractical with 90+ tasks across 9 feature areas
- **Deployment velocity** slows to a crawl as teams fear breaking production
- **Rollback procedures** become ad-hoc and unreliable during incidents
- **Environment drift** makes debugging nearly impossible when issues only appear in production

This pipeline enables the team to ship enterprise features **confidently and frequently**, with the safety nets required for production-grade software. It's the infrastructure foundation that makes all other enterprise tasks viable in production.

## Integration Context

**Upstream Dependencies:**

This task depends on **Task 81** (CI/CD Quality Gates) which establishes:
- Test coverage requirements (>90%)
- PHPStan level 5 with zero errors
- Security scanning integration
- Performance benchmarking infrastructure

These quality gates are enforced at each deployment stage in the multi-environment pipeline.

**Integration Points:**

1. **Testing Infrastructure (Task 81)**: Quality gates run before each environment deployment
2. **Database Migrations (Task 90)**: Migration validation integrated into deployment workflow
3. **Terraform Infrastructure**: Environment-specific infrastructure provisioned via Terraform
4. **Monitoring & Alerting (Task 91)**: Post-deployment health checks and alert routing
5. **Organization Data**: Seeding and migration of organization hierarchy in non-production environments

**Downstream Impact:**

- **All feature deployments**: Every task (2-88) benefits from automated deployment pipeline
- **Developer productivity**: Faster feedback loops with automated dev deployments
- **Production stability**: Reduced deployment risk through staged rollouts
- **Incident response**: Faster rollbacks with automated reversion capabilities

## Acceptance Criteria

- [ ] Three distinct environments configured: development, staging, production
- [ ] GitHub Actions workflow (or equivalent) created for automated deployments
- [ ] Environment-specific configuration management using Laravel's environment files
- [ ] Automated deployment to development environment on every main branch commit
- [ ] Automated deployment to staging environment on successful development deployment
- [ ] Manual approval required for production deployments
- [ ] Database migration validation with automatic rollback on failure
- [ ] Zero-downtime deployment strategy implemented (blue-green or rolling)
- [ ] Automated health checks after each deployment
- [ ] Automatic rollback on failed health checks
- [ ] Environment-specific Docker images with appropriate tags
- [ ] Secret management integrated with environment variables
- [ ] Slack/Discord notification on deployment success/failure
- [ ] Deployment status dashboard accessible to team
- [ ] Database backup created automatically before production deployments
- [ ] Application logs centralized and searchable
- [ ] Performance benchmarks compared against baseline after staging deployments
- [ ] Security scanning integrated into pipeline (dependency vulnerabilities, code analysis)

## Technical Details

### File Paths

**CI/CD Configuration:**
- `/home/topgun/topgun/.github/workflows/deploy-dev.yml` (new)
- `/home/topgun/topgun/.github/workflows/deploy-staging.yml` (new)
- `/home/topgun/topgun/.github/workflows/deploy-production.yml` (new)
- `/home/topgun/topgun/.github/workflows/quality-gates.yml` (new - reusable workflow)

**Environment Configuration:**
- `/home/topgun/topgun/.env.dev.example` (new)
- `/home/topgun/topgun/.env.staging.example` (new)
- `/home/topgun/topgun/.env.production.example` (new)

**Deployment Scripts:**
- `/home/topgun/topgun/scripts/deploy/pre-deploy-checks.sh` (new)
- `/home/topgun/topgun/scripts/deploy/migrate-with-validation.sh` (new)
- `/home/topgun/topgun/scripts/deploy/health-check.sh` (new)
- `/home/topgun/topgun/scripts/deploy/rollback.sh` (new)
- `/home/topgun/topgun/scripts/deploy/backup-database.sh` (new)

**Docker:**
- `/home/topgun/topgun/Dockerfile.production` (new - optimized production image)
- `/home/topgun/topgun/docker-compose.production.yml` (new)
- `/home/topgun/topgun/.dockerignore` (modify)

**Terraform:**
- `/home/topgun/topgun/infrastructure/environments/dev/main.tf` (new)
- `/home/topgun/topgun/infrastructure/environments/staging/main.tf` (new)
- `/home/topgun/topgun/infrastructure/environments/production/main.tf` (new)

### GitHub Actions Workflow Structure

#### Development Deployment Workflow

**File:** `.github/workflows/deploy-dev.yml`

```yaml
name: Deploy to Development

on:
  push:
    branches:
      - main
  workflow_dispatch:

env:
  ENVIRONMENT: development
  DOCKER_REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}/coolify-enterprise

jobs:
  quality-gates:
    uses: ./.github/workflows/quality-gates.yml
    secrets: inherit

  build:
    needs: quality-gates
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.DOCKER_REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata for Docker
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=ref,event=branch
            type=sha,prefix=dev-
            type=raw,value=dev-latest

      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile.production
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache-dev
          cache-to: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache-dev,mode=max
          build-args: |
            APP_ENV=development

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: development
      url: https://dev.coolify-enterprise.example.com

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup SSH
        uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.DEV_SSH_PRIVATE_KEY }}

      - name: Pre-deployment checks
        run: |
          chmod +x scripts/deploy/pre-deploy-checks.sh
          ./scripts/deploy/pre-deploy-checks.sh
        env:
          DEPLOY_ENV: development
          SSH_HOST: ${{ secrets.DEV_SSH_HOST }}
          SSH_USER: ${{ secrets.DEV_SSH_USER }}

      - name: Backup database
        run: |
          chmod +x scripts/deploy/backup-database.sh
          ./scripts/deploy/backup-database.sh
        env:
          DB_HOST: ${{ secrets.DEV_DB_HOST }}
          DB_NAME: ${{ secrets.DEV_DB_NAME }}
          DB_USER: ${{ secrets.DEV_DB_USER }}
          DB_PASSWORD: ${{ secrets.DEV_DB_PASSWORD }}
          BACKUP_BUCKET: ${{ secrets.DEV_BACKUP_BUCKET }}

      - name: Pull new Docker image
        run: |
          ssh ${{ secrets.DEV_SSH_USER }}@${{ secrets.DEV_SSH_HOST }} \
            "docker pull ${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}:dev-latest"

      - name: Run database migrations with validation
        run: |
          chmod +x scripts/deploy/migrate-with-validation.sh
          ./scripts/deploy/migrate-with-validation.sh
        env:
          SSH_HOST: ${{ secrets.DEV_SSH_HOST }}
          SSH_USER: ${{ secrets.DEV_SSH_USER }}
          CONTAINER_NAME: coolify-enterprise-dev

      - name: Deploy application (rolling update)
        run: |
          ssh ${{ secrets.DEV_SSH_USER }}@${{ secrets.DEV_SSH_HOST }} << 'ENDSSH'
            cd /opt/coolify-enterprise

            # Pull latest configuration
            git pull origin main

            # Update environment variables
            cp .env.dev .env

            # Rolling update with Docker Compose
            docker-compose -f docker-compose.production.yml up -d --force-recreate --remove-orphans

            # Wait for containers to be healthy
            sleep 10
          ENDSSH

      - name: Health check
        run: |
          chmod +x scripts/deploy/health-check.sh
          ./scripts/deploy/health-check.sh
        env:
          APP_URL: https://dev.coolify-enterprise.example.com
          HEALTH_ENDPOINT: /api/health
          MAX_RETRIES: 10
          RETRY_DELAY: 10

      - name: Rollback on failure
        if: failure()
        run: |
          chmod +x scripts/deploy/rollback.sh
          ./scripts/deploy/rollback.sh
        env:
          SSH_HOST: ${{ secrets.DEV_SSH_HOST }}
          SSH_USER: ${{ secrets.DEV_SSH_USER }}
          BACKUP_TAG: ${{ github.sha }}-pre-deploy

      - name: Notify deployment status
        if: always()
        uses: slackapi/slack-github-action@v1.25.0
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "Development Deployment ${{ job.status }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Development Deployment*\nStatus: ${{ job.status }}\nCommit: `${{ github.sha }}`\nBranch: `${{ github.ref_name }}`\nActor: ${{ github.actor }}"
                  }
                }
              ]
            }
```

#### Staging Deployment Workflow

**File:** `.github/workflows/deploy-staging.yml`

```yaml
name: Deploy to Staging

on:
  workflow_run:
    workflows: ["Deploy to Development"]
    types:
      - completed
    branches:
      - main
  workflow_dispatch:

env:
  ENVIRONMENT: staging
  DOCKER_REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}/coolify-enterprise

jobs:
  check-dev-deployment:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - name: Verify development deployment succeeded
        run: echo "Development deployment was successful"

  quality-gates:
    needs: check-dev-deployment
    uses: ./.github/workflows/quality-gates.yml
    secrets: inherit
    with:
      run-performance-tests: true

  build:
    needs: quality-gates
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.DOCKER_REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata for Docker
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=ref,event=branch
            type=sha,prefix=staging-
            type=raw,value=staging-latest

      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile.production
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache-staging
          cache-to: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache-staging,mode=max
          build-args: |
            APP_ENV=staging

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: staging
      url: https://staging.coolify-enterprise.example.com

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup SSH
        uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.STAGING_SSH_PRIVATE_KEY }}

      # Similar deployment steps as dev, but with staging-specific configurations
      - name: Pre-deployment checks
        run: |
          chmod +x scripts/deploy/pre-deploy-checks.sh
          ./scripts/deploy/pre-deploy-checks.sh
        env:
          DEPLOY_ENV: staging
          SSH_HOST: ${{ secrets.STAGING_SSH_HOST }}
          SSH_USER: ${{ secrets.STAGING_SSH_USER }}

      - name: Backup database
        run: |
          chmod +x scripts/deploy/backup-database.sh
          ./scripts/deploy/backup-database.sh
        env:
          DB_HOST: ${{ secrets.STAGING_DB_HOST }}
          DB_NAME: ${{ secrets.STAGING_DB_NAME }}
          DB_USER: ${{ secrets.STAGING_DB_USER }}
          DB_PASSWORD: ${{ secrets.STAGING_DB_PASSWORD }}
          BACKUP_BUCKET: ${{ secrets.STAGING_BACKUP_BUCKET }}

      - name: Deploy application (blue-green deployment)
        run: |
          ssh ${{ secrets.STAGING_SSH_USER }}@${{ secrets.STAGING_SSH_HOST }} << 'ENDSSH'
            cd /opt/coolify-enterprise

            # Blue-green deployment strategy
            # 1. Start new "green" containers
            docker-compose -f docker-compose.production.yml -p coolify-green up -d

            # 2. Wait for green to be healthy
            sleep 30

            # 3. Run health checks on green
            curl -f http://localhost:8001/api/health || exit 1

            # 4. Switch load balancer to green
            docker exec nginx-lb /scripts/switch-to-green.sh

            # 5. Wait for traffic to drain from blue
            sleep 10

            # 6. Stop blue containers
            docker-compose -f docker-compose.production.yml -p coolify-blue down

            # 7. Rename green to blue for next deployment
            docker-compose -f docker-compose.production.yml -p coolify-green stop
            docker rename coolify-green coolify-blue
          ENDSSH

      - name: Run smoke tests
        run: |
          npm run test:smoke -- --env=staging
        env:
          STAGING_URL: https://staging.coolify-enterprise.example.com
          STAGING_API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}

      - name: Performance benchmark comparison
        run: |
          npm run benchmark:compare -- --env=staging --baseline=v1.0.0

      - name: Health check
        run: |
          chmod +x scripts/deploy/health-check.sh
          ./scripts/deploy/health-check.sh
        env:
          APP_URL: https://staging.coolify-enterprise.example.com
          HEALTH_ENDPOINT: /api/health
          MAX_RETRIES: 10
          RETRY_DELAY: 10

      - name: Rollback on failure
        if: failure()
        run: |
          chmod +x scripts/deploy/rollback.sh
          ./scripts/deploy/rollback.sh
        env:
          SSH_HOST: ${{ secrets.STAGING_SSH_HOST }}
          SSH_USER: ${{ secrets.STAGING_SSH_USER }}
          DEPLOYMENT_STRATEGY: blue-green

      - name: Notify deployment status
        if: always()
        uses: slackapi/slack-github-action@v1.25.0
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "Staging Deployment ${{ job.status }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Staging Deployment*\nStatus: ${{ job.status }}\nCommit: `${{ github.sha }}`\nEnvironment: staging\nURL: https://staging.coolify-enterprise.example.com"
                  }
                }
              ]
            }
```

#### Production Deployment Workflow

**File:** `.github/workflows/deploy-production.yml`

```yaml
name: Deploy to Production

on:
  workflow_dispatch:
    inputs:
      deployment_strategy:
        description: 'Deployment strategy'
        required: true
        default: 'blue-green'
        type: choice
        options:
          - blue-green
          - rolling
          - canary

env:
  ENVIRONMENT: production
  DOCKER_REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}/coolify-enterprise

jobs:
  quality-gates:
    uses: ./.github/workflows/quality-gates.yml
    secrets: inherit
    with:
      run-performance-tests: true
      run-security-scan: true
      run-load-tests: true

  build:
    needs: quality-gates
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.DOCKER_REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata for Docker
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=semver,pattern={{version}}
            type=semver,pattern={{major}}.{{minor}}
            type=sha,prefix=prod-
            type=raw,value=production-latest

      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile.production
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache-prod
          cache-to: type=registry,ref=${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}:buildcache-prod,mode=max
          build-args: |
            APP_ENV=production

  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://coolify-enterprise.example.com

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Require manual approval
        uses: trstringer/manual-approval@v1
        with:
          secret: ${{ secrets.GITHUB_TOKEN }}
          approvers: tech-lead,devops-lead,cto
          minimum-approvals: 2
          issue-title: "Production Deployment Approval Required"
          issue-body: |
            **Production Deployment Request**

            Commit: ${{ github.sha }}
            Branch: ${{ github.ref_name }}
            Actor: ${{ github.actor }}
            Strategy: ${{ github.event.inputs.deployment_strategy }}

            Please review and approve/reject this production deployment.

      - name: Setup SSH
        uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.PROD_SSH_PRIVATE_KEY }}

      - name: Pre-deployment checks
        run: |
          chmod +x scripts/deploy/pre-deploy-checks.sh
          ./scripts/deploy/pre-deploy-checks.sh
        env:
          DEPLOY_ENV: production
          SSH_HOST: ${{ secrets.PROD_SSH_HOST }}
          SSH_USER: ${{ secrets.PROD_SSH_USER }}

      - name: Create production database backup
        run: |
          chmod +x scripts/deploy/backup-database.sh
          ./scripts/deploy/backup-database.sh
        env:
          DB_HOST: ${{ secrets.PROD_DB_HOST }}
          DB_NAME: ${{ secrets.PROD_DB_NAME }}
          DB_USER: ${{ secrets.PROD_DB_USER }}
          DB_PASSWORD: ${{ secrets.PROD_DB_PASSWORD }}
          BACKUP_BUCKET: ${{ secrets.PROD_BACKUP_BUCKET }}
          BACKUP_RETENTION_DAYS: 90

      - name: Enable maintenance mode
        run: |
          ssh ${{ secrets.PROD_SSH_USER }}@${{ secrets.PROD_SSH_HOST }} \
            "cd /opt/coolify-enterprise && docker-compose exec app php artisan down --retry=60"

      - name: Run database migrations with validation
        run: |
          chmod +x scripts/deploy/migrate-with-validation.sh
          ./scripts/deploy/migrate-with-validation.sh
        env:
          SSH_HOST: ${{ secrets.PROD_SSH_HOST }}
          SSH_USER: ${{ secrets.PROD_SSH_USER }}
          CONTAINER_NAME: coolify-enterprise-prod
          MIGRATION_TIMEOUT: 600

      - name: Deploy application
        run: |
          case "${{ github.event.inputs.deployment_strategy }}" in
            blue-green)
              chmod +x scripts/deploy/deploy-blue-green.sh
              ./scripts/deploy/deploy-blue-green.sh
              ;;
            rolling)
              chmod +x scripts/deploy/deploy-rolling.sh
              ./scripts/deploy/deploy-rolling.sh
              ;;
            canary)
              chmod +x scripts/deploy/deploy-canary.sh
              ./scripts/deploy/deploy-canary.sh
              ;;
          esac
        env:
          SSH_HOST: ${{ secrets.PROD_SSH_HOST }}
          SSH_USER: ${{ secrets.PROD_SSH_USER }}
          DOCKER_IMAGE: ${{ env.DOCKER_REGISTRY }}/${{ env.IMAGE_NAME }}:production-latest

      - name: Disable maintenance mode
        run: |
          ssh ${{ secrets.PROD_SSH_USER }}@${{ secrets.PROD_SSH_HOST }} \
            "cd /opt/coolify-enterprise && docker-compose exec app php artisan up"

      - name: Health check
        run: |
          chmod +x scripts/deploy/health-check.sh
          ./scripts/deploy/health-check.sh
        env:
          APP_URL: https://coolify-enterprise.example.com
          HEALTH_ENDPOINT: /api/health
          MAX_RETRIES: 20
          RETRY_DELAY: 15

      - name: Run smoke tests
        run: |
          npm run test:smoke -- --env=production
        env:
          PROD_URL: https://coolify-enterprise.example.com
          PROD_API_TOKEN: ${{ secrets.PROD_API_TOKEN }}

      - name: Monitor deployment metrics
        run: |
          # Wait 5 minutes and check error rates, response times
          sleep 300
          npm run metrics:check -- --env=production --threshold=0.01

      - name: Rollback on failure
        if: failure()
        run: |
          chmod +x scripts/deploy/rollback.sh
          ./scripts/deploy/rollback.sh
        env:
          SSH_HOST: ${{ secrets.PROD_SSH_HOST }}
          SSH_USER: ${{ secrets.PROD_SSH_USER }}
          DEPLOYMENT_STRATEGY: ${{ github.event.inputs.deployment_strategy }}
          NOTIFY_SLACK: true

      - name: Tag successful deployment
        if: success()
        run: |
          git tag -a "prod-${{ github.sha }}" -m "Production deployment successful"
          git push origin "prod-${{ github.sha }}"

      - name: Notify deployment status
        if: always()
        uses: slackapi/slack-github-action@v1.25.0
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK_URL }}
          payload: |
            {
              "text": "🚀 Production Deployment ${{ job.status }}",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Production Deployment*\nStatus: ${{ job.status }}\nStrategy: ${{ github.event.inputs.deployment_strategy }}\nCommit: `${{ github.sha }}`\nActor: ${{ github.actor }}\nURL: https://coolify-enterprise.example.com"
                  }
                },
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "${{ job.status == 'success' && '✅ Deployment completed successfully' || '❌ Deployment failed - rollback initiated' }}"
                  }
                }
              ]
            }
```

#### Reusable Quality Gates Workflow

**File:** `.github/workflows/quality-gates.yml`

```yaml
name: Quality Gates

on:
  workflow_call:
    inputs:
      run-performance-tests:
        type: boolean
        default: false
      run-security-scan:
        type: boolean
        default: false
      run-load-tests:
        type: boolean
        default: false

jobs:
  phpstan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: '8.4'
          extensions: mbstring, dom, fileinfo, pgsql
          coverage: none

      - name: Install dependencies
        run: composer install --prefer-dist --no-progress

      - name: Run PHPStan
        run: ./vendor/bin/phpstan analyse --level=5 --no-progress --error-format=github

  pint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: '8.4'

      - name: Install dependencies
        run: composer install --prefer-dist --no-progress

      - name: Run Laravel Pint
        run: ./vendor/bin/pint --test

  pest:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_DB: coolify_test
          POSTGRES_USER: coolify
          POSTGRES_PASSWORD: secret
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432

      redis:
        image: redis:7
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 6379:6379

    steps:
      - uses: actions/checkout@v4

      - name: Setup PHP
        uses: shivammathur/setup-php@v2
        with:
          php-version: '8.4'
          extensions: mbstring, dom, fileinfo, pgsql, redis
          coverage: xdebug

      - name: Install dependencies
        run: composer install --prefer-dist --no-progress

      - name: Prepare Laravel application
        run: |
          cp .env.ci .env
          php artisan key:generate

      - name: Run migrations
        run: php artisan migrate --force

      - name: Run Pest tests with coverage
        run: ./vendor/bin/pest --coverage --min=90

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage.xml
          flags: php

  vue-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run Vitest
        run: npm run test:coverage

      - name: Upload coverage
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage/coverage-final.json
          flags: vue

  security-scan:
    if: ${{ inputs.run-security-scan }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-results.sarif'

      - name: Upload Trivy results to GitHub Security
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'trivy-results.sarif'

  performance-tests:
    if: ${{ inputs.run-performance-tests }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Run Lighthouse CI
        run: |
          npm install -g @lhci/cli
          lhci autorun --config=.lighthouserc.json
```

### Deployment Scripts

#### Pre-Deployment Checks Script

**File:** `scripts/deploy/pre-deploy-checks.sh`

```bash
#!/bin/bash
set -euo pipefail

DEPLOY_ENV=${DEPLOY_ENV:-development}
SSH_HOST=${SSH_HOST}
SSH_USER=${SSH_USER}

echo "🔍 Running pre-deployment checks for ${DEPLOY_ENV}..."

# Check server connectivity
echo "Checking SSH connectivity..."
ssh -o ConnectTimeout=10 ${SSH_USER}@${SSH_HOST} "echo 'SSH connection successful'"

# Check disk space
echo "Checking disk space..."
DISK_USAGE=$(ssh ${SSH_USER}@${SSH_HOST} "df -h / | tail -1 | awk '{print \$5}' | sed 's/%//'")
if [ "$DISK_USAGE" -gt 85 ]; then
    echo "❌ ERROR: Disk usage is ${DISK_USAGE}% - deployment aborted"
    exit 1
fi

# Check Docker daemon
echo "Checking Docker daemon..."
ssh ${SSH_USER}@${SSH_HOST} "docker info > /dev/null 2>&1" || {
    echo "❌ ERROR: Docker daemon not running"
    exit 1
}

# Check database connectivity
echo "Checking database connectivity..."
ssh ${SSH_USER}@${SSH_HOST} "docker-compose exec -T db pg_isready -U coolify" || {
    echo "❌ ERROR: Database not accessible"
    exit 1
}

# Check Redis connectivity
echo "Checking Redis connectivity..."
ssh ${SSH_USER}@${SSH_HOST} "docker-compose exec -T redis redis-cli ping" || {
    echo "❌ ERROR: Redis not accessible"
    exit 1
}

echo "✅ All pre-deployment checks passed"
```

#### Migration Validation Script

**File:** `scripts/deploy/migrate-with-validation.sh`

```bash
#!/bin/bash
set -euo pipefail

SSH_HOST=${SSH_HOST}
SSH_USER=${SSH_USER}
CONTAINER_NAME=${CONTAINER_NAME:-coolify-enterprise}
MIGRATION_TIMEOUT=${MIGRATION_TIMEOUT:-300}

echo "🔄 Running database migrations with validation..."

# Test migrations in dry-run mode first
echo "Testing migrations (dry-run)..."
ssh ${SSH_USER}@${SSH_HOST} << ENDSSH
    docker exec ${CONTAINER_NAME} php artisan migrate:status

    # Create migration backup point
    docker exec ${CONTAINER_NAME} php artisan db:backup-schema --tag=pre-migration
ENDSSH

# Run actual migrations
echo "Running migrations..."
ssh ${SSH_USER}@${SSH_HOST} << ENDSSH
    timeout ${MIGRATION_TIMEOUT} docker exec ${CONTAINER_NAME} php artisan migrate --force || {
        echo "❌ Migration failed - attempting rollback"
        docker exec ${CONTAINER_NAME} php artisan migrate:rollback --force
        docker exec ${CONTAINER_NAME} php artisan db:restore-schema --tag=pre-migration
        exit 1
    }
ENDSSH

# Validate migration success
echo "Validating migrations..."
ssh ${SSH_USER}@${SSH_HOST} << ENDSSH
    docker exec ${CONTAINER_NAME} php artisan migrate:status | grep -q "Ran" || {
        echo "❌ Migration validation failed"
        exit 1
    }
ENDSSH

echo "✅ Migrations completed successfully"
```

#### Health Check Script

**File:** `scripts/deploy/health-check.sh`

```bash
#!/bin/bash
set -euo pipefail

APP_URL=${APP_URL}
HEALTH_ENDPOINT=${HEALTH_ENDPOINT:-/api/health}
MAX_RETRIES=${MAX_RETRIES:-10}
RETRY_DELAY=${RETRY_DELAY:-10}

echo "🏥 Running health checks against ${APP_URL}${HEALTH_ENDPOINT}..."

for i in $(seq 1 ${MAX_RETRIES}); do
    echo "Health check attempt $i/${MAX_RETRIES}..."

    HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "${APP_URL}${HEALTH_ENDPOINT}" || echo "000")

    if [ "$HTTP_STATUS" = "200" ]; then
        echo "✅ Health check passed (HTTP 200)"

        # Additional checks
        RESPONSE=$(curl -s "${APP_URL}${HEALTH_ENDPOINT}")

        # Check database connectivity
        echo "$RESPONSE" | jq -e '.database == "healthy"' > /dev/null || {
            echo "❌ Database health check failed"
            exit 1
        }

        # Check Redis connectivity
        echo "$RESPONSE" | jq -e '.redis == "healthy"' > /dev/null || {
            echo "❌ Redis health check failed"
            exit 1
        }

        echo "✅ All health checks passed"
        exit 0
    fi

    echo "Health check returned HTTP ${HTTP_STATUS}, retrying in ${RETRY_DELAY}s..."
    sleep ${RETRY_DELAY}
done

echo "❌ Health checks failed after ${MAX_RETRIES} attempts"
exit 1
```

#### Database Backup Script

**File:** `scripts/deploy/backup-database.sh`

```bash
#!/bin/bash
set -euo pipefail

DB_HOST=${DB_HOST}
DB_NAME=${DB_NAME}
DB_USER=${DB_USER}
DB_PASSWORD=${DB_PASSWORD}
BACKUP_BUCKET=${BACKUP_BUCKET}
BACKUP_RETENTION_DAYS=${BACKUP_RETENTION_DAYS:-30}

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="${DB_NAME}_${TIMESTAMP}.sql.gz"

echo "📦 Creating database backup: ${BACKUP_FILE}..."

# Create backup
PGPASSWORD=${DB_PASSWORD} pg_dump -h ${DB_HOST} -U ${DB_USER} ${DB_NAME} | gzip > /tmp/${BACKUP_FILE}

# Upload to S3-compatible storage
aws s3 cp /tmp/${BACKUP_FILE} s3://${BACKUP_BUCKET}/backups/${BACKUP_FILE}

# Clean up old backups
aws s3 ls s3://${BACKUP_BUCKET}/backups/ | \
    awk '{print $4}' | \
    head -n -${BACKUP_RETENTION_DAYS} | \
    xargs -I {} aws s3 rm s3://${BACKUP_BUCKET}/backups/{}

# Clean up local file
rm /tmp/${BACKUP_FILE}

echo "✅ Backup created and uploaded successfully"
echo "Backup location: s3://${BACKUP_BUCKET}/backups/${BACKUP_FILE}"
```

#### Rollback Script

**File:** `scripts/deploy/rollback.sh`

```bash
#!/bin/bash
set -euo pipefail

SSH_HOST=${SSH_HOST}
SSH_USER=${SSH_USER}
DEPLOYMENT_STRATEGY=${DEPLOYMENT_STRATEGY:-rolling}
NOTIFY_SLACK=${NOTIFY_SLACK:-false}

echo "🔙 Initiating rollback (strategy: ${DEPLOYMENT_STRATEGY})..."

case "${DEPLOYMENT_STRATEGY}" in
    blue-green)
        echo "Rolling back blue-green deployment..."
        ssh ${SSH_USER}@${SSH_HOST} << 'ENDSSH'
            cd /opt/coolify-enterprise

            # Switch load balancer back to blue
            docker exec nginx-lb /scripts/switch-to-blue.sh

            # Stop green containers
            docker-compose -p coolify-green down

            # Restart blue containers if needed
            docker-compose -p coolify-blue up -d
        ENDSSH
        ;;

    rolling)
        echo "Rolling back rolling deployment..."
        ssh ${SSH_USER}@${SSH_HOST} << 'ENDSSH'
            cd /opt/coolify-enterprise

            # Pull previous image tag
            docker pull ghcr.io/coolify/enterprise:previous

            # Force recreate with previous image
            docker-compose up -d --force-recreate
        ENDSSH
        ;;

    canary)
        echo "Rolling back canary deployment..."
        ssh ${SSH_USER}@${SSH_HOST} << 'ENDSSH'
            cd /opt/coolify-enterprise

            # Set traffic routing to 100% stable
            docker exec nginx-lb /scripts/route-to-stable.sh

            # Stop canary containers
            docker-compose -p coolify-canary down
        ENDSSH
        ;;
esac

# Rollback database migrations
echo "Rolling back database migrations..."
ssh ${SSH_USER}@${SSH_HOST} << 'ENDSSH'
    docker exec coolify-enterprise php artisan migrate:rollback --force --step=1
ENDSSH

echo "✅ Rollback completed"

if [ "$NOTIFY_SLACK" = "true" ]; then
    curl -X POST ${SLACK_WEBHOOK_URL} -H 'Content-Type: application/json' -d '{
        "text": "🔙 Deployment rolled back",
        "blocks": [{
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "*Rollback Completed*\nStrategy: '"${DEPLOYMENT_STRATEGY}"'\nEnvironment: '"${SSH_HOST}"'"
            }
        }]
    }'
fi
```

### Production-Optimized Dockerfile

**File:** `Dockerfile.production`

```dockerfile
# Build stage
FROM node:20-alpine AS node-builder

WORKDIR /app

COPY package*.json ./
RUN npm ci --production=false

COPY resources/ resources/
COPY vite.config.js ./
COPY tailwind.config.js ./
COPY postcss.config.js ./

RUN npm run build

# PHP stage
FROM php:8.4-fpm-alpine

# Install system dependencies
RUN apk add --no-cache \
    postgresql-dev \
    libzip-dev \
    icu-dev \
    oniguruma-dev \
    supervisor \
    nginx \
    && docker-php-ext-install \
        pdo_pgsql \
        zip \
        intl \
        opcache \
        pcntl

# Install Composer
COPY --from=composer:2 /usr/bin/composer /usr/bin/composer

# Configure PHP for production
RUN mv "$PHP_INI_DIR/php.ini-production" "$PHP_INI_DIR/php.ini"
COPY docker/php/opcache.ini $PHP_INI_DIR/conf.d/opcache.ini

WORKDIR /var/www/html

# Copy application
COPY --chown=www-data:www-data . .
COPY --from=node-builder --chown=www-data:www-data /app/public/build ./public/build

# Install PHP dependencies
RUN composer install --no-dev --optimize-autoloader --no-interaction

# Optimize Laravel
RUN php artisan config:cache && \
    php artisan route:cache && \
    php artisan view:cache

# Configure supervisord
COPY docker/supervisord.conf /etc/supervisor/conf.d/supervisord.conf

# Configure Nginx
COPY docker/nginx/default.conf /etc/nginx/http.d/default.conf

EXPOSE 80

CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
```

### Environment Configuration Examples

**File:** `.env.dev.example`

```bash
APP_NAME="Coolify Enterprise (Dev)"
APP_ENV=development
APP_DEBUG=true
APP_URL=https://dev.coolify-enterprise.example.com

DB_CONNECTION=pgsql
DB_HOST=dev-db.internal
DB_PORT=5432
DB_DATABASE=coolify_dev
DB_USERNAME=coolify
DB_PASSWORD=

REDIS_HOST=dev-redis.internal
REDIS_PASSWORD=
REDIS_PORT=6379

CACHE_DRIVER=redis
SESSION_DRIVER=redis
QUEUE_CONNECTION=redis

# Terraform (use test credentials)
TERRAFORM_BINARY_PATH=/usr/local/bin/terraform

# Feature Flags
ENABLE_TERRAFORM=true
ENABLE_PAYMENT_PROCESSING=false
```

**File:** `.env.staging.example`

```bash
APP_NAME="Coolify Enterprise (Staging)"
APP_ENV=staging
APP_DEBUG=false
APP_URL=https://staging.coolify-enterprise.example.com

DB_CONNECTION=pgsql
DB_HOST=staging-db.internal
DB_PORT=5432
DB_DATABASE=coolify_staging
DB_USERNAME=coolify
DB_PASSWORD=

REDIS_HOST=staging-redis.internal
REDIS_PASSWORD=
REDIS_PORT=6379

CACHE_DRIVER=redis
SESSION_DRIVER=redis
QUEUE_CONNECTION=redis

# Terraform
TERRAFORM_BINARY_PATH=/usr/local/bin/terraform

# Feature Flags
ENABLE_TERRAFORM=true
ENABLE_PAYMENT_PROCESSING=true
```

**File:** `.env.production.example`

```bash
APP_NAME="Coolify Enterprise"
APP_ENV=production
APP_DEBUG=false
APP_URL=https://coolify-enterprise.example.com

DB_CONNECTION=pgsql
DB_HOST=prod-db.internal
DB_PORT=5432
DB_DATABASE=coolify_production
DB_USERNAME=coolify
DB_PASSWORD=

REDIS_HOST=prod-redis.internal
REDIS_PASSWORD=
REDIS_PORT=6379

CACHE_DRIVER=redis
SESSION_DRIVER=redis
QUEUE_CONNECTION=redis

# Logging
LOG_CHANNEL=stack
LOG_DEPRECATIONS_CHANNEL=null
LOG_LEVEL=warning

# Performance
OCTANE_SERVER=swoole

# Terraform
TERRAFORM_BINARY_PATH=/usr/local/bin/terraform

# Feature Flags
ENABLE_TERRAFORM=true
ENABLE_PAYMENT_PROCESSING=true
```

## Implementation Approach

### Step 1: Environment Infrastructure Setup (3-4 hours)
1. Provision development, staging, and production servers (can use Terraform)
2. Install Docker, Docker Compose, and required dependencies on each server
3. Configure firewall rules, SSH access, and network security groups
4. Set up PostgreSQL and Redis instances for each environment

### Step 2: GitHub Actions Configuration (2-3 hours)
1. Create `.github/workflows/` directory structure
2. Implement `quality-gates.yml` reusable workflow
3. Create `deploy-dev.yml`, `deploy-staging.yml`, `deploy-production.yml`
4. Configure GitHub Secrets for each environment (SSH keys, DB credentials, etc.)
5. Set up GitHub Environments with protection rules

### Step 3: Deployment Scripts (3-4 hours)
1. Create `scripts/deploy/` directory
2. Implement `pre-deploy-checks.sh` with validation logic
3. Implement `migrate-with-validation.sh` with rollback capability
4. Implement `health-check.sh` with comprehensive checks
5. Implement `backup-database.sh` with S3 integration
6. Implement `rollback.sh` for each deployment strategy
7. Make all scripts executable: `chmod +x scripts/deploy/*.sh`

### Step 4: Docker Production Optimization (2-3 hours)
1. Create `Dockerfile.production` with multi-stage build
2. Optimize PHP configuration for production (OPcache, memory limits)
3. Create `docker-compose.production.yml` with production services
4. Configure Nginx for reverse proxy and static asset serving
5. Set up Supervisor for queue workers and schedule

### Step 5: Environment Configuration (1-2 hours)
1. Create `.env.dev.example`, `.env.staging.example`, `.env.production.example`
2. Document all required environment variables
3. Configure Laravel environment-specific settings
4. Set up feature flags for gradual rollouts

### Step 6: Database Migration Safety (2-3 hours)
1. Enhance migration system with validation hooks
2. Implement schema backup before migrations
3. Add migration timeout handling
4. Create migration rollback automation
5. Test with complex migration scenarios

### Step 7: Monitoring & Alerting Integration (2-3 hours)
1. Create `/api/health` endpoint with comprehensive checks
2. Integrate Slack notifications for deployment events
3. Set up error tracking (Sentry/Bugsnag) for each environment
4. Configure application performance monitoring (New Relic/DataDog)
5. Create deployment dashboard for status visibility

### Step 8: Testing & Validation (3-4 hours)
1. Write smoke tests for critical user journeys
2. Create performance benchmark tests
3. Test deployment workflows in development environment
4. Validate rollback procedures work correctly
5. Document deployment runbooks

### Step 9: Security & Compliance (1-2 hours)
1. Enable security scanning in CI/CD pipeline
2. Configure secret scanning for sensitive data
3. Set up dependency vulnerability scanning
4. Implement RBAC for production deployments
5. Document compliance procedures

### Step 10: Documentation & Training (2-3 hours)
1. Write deployment runbook for team
2. Document emergency rollback procedures
3. Create troubleshooting guide
4. Train team on new deployment process
5. Conduct dry-run deployment with team

## Test Strategy

### Unit Tests

Test individual deployment scripts in isolation:

**File:** `tests/Unit/Deployment/DeploymentScriptsTest.php`

```php
<?php

it('validates pre-deployment checks script exists and is executable', function () {
    $scriptPath = base_path('scripts/deploy/pre-deploy-checks.sh');

    expect(file_exists($scriptPath))->toBeTrue();
    expect(is_executable($scriptPath))->toBeTrue();
});

it('validates health check script can parse JSON response', function () {
    // Mock health endpoint response
    $healthResponse = json_encode([
        'status' => 'healthy',
        'database' => 'healthy',
        'redis' => 'healthy',
    ]);

    // Test script logic (extracted to PHP class for testing)
    $healthCheck = new \App\Services\Deployment\HealthCheckService();
    $result = $healthCheck->validateResponse($healthResponse);

    expect($result)->toBeTrue();
});

it('validates migration rollback logic', function () {
    // Test migration rollback service
    $migrationService = app(\App\Services\Deployment\MigrationService::class);

    // Run migration
    $migrationService->migrate();

    // Rollback
    $result = $migrationService->rollback();

    expect($result)->toBeTrue();
});
```

### Integration Tests

Test complete deployment workflows:

**File:** `tests/Feature/Deployment/DeploymentWorkflowTest.php`

```php
<?php

use Illuminate\Support\Facades\Artisan;

it('performs health check endpoint correctly', function () {
    $response = $this->get('/api/health');

    $response->assertOk()
        ->assertJson([
            'status' => 'healthy',
            'database' => 'healthy',
            'redis' => 'healthy',
            'version' => config('app.version'),
        ]);
});

it('validates database migration with rollback capability', function () {
    // Create test migration
    Artisan::call('make:migration', ['name' => 'test_rollback_migration']);

    // Run migration
    Artisan::call('migrate', ['--force' => true]);

    // Verify migration ran
    expect(Schema::hasTable('test_table'))->toBeTrue();

    // Rollback
    Artisan::call('migrate:rollback', ['--force' => true, '--step' => 1]);

    // Verify rollback worked
    expect(Schema::hasTable('test_table'))->toBeFalse();
});

it('maintains application availability during rolling deployment', function () {
    // Simulate rolling deployment with health checks
    $healthCheckService = app(\App\Services\Deployment\HealthCheckService::class);

    // Initial health check
    expect($healthCheckService->check())->toBeTrue();

    // Simulate deployment (restart queue workers, clear cache)
    Artisan::call('cache:clear');
    Artisan::call('queue:restart');

    // Health check should still pass
    expect($healthCheckService->check())->toBeTrue();
});
```

### Smoke Tests

Test critical user journeys after deployment:

**File:** `tests/Smoke/CriticalJourneysTest.js`

```javascript
import { test, expect } from '@playwright/test';

test.describe('Critical User Journeys (Smoke Tests)', () => {
  test('user can log in', async ({ page }) => {
    await page.goto(process.env.APP_URL);
    await page.fill('input[name="email"]', 'admin@example.com');
    await page.fill('input[name="password"]', 'password');
    await page.click('button[type="submit"]');

    await expect(page).toHaveURL(/.*\/dashboard/);
  });

  test('user can view organization dashboard', async ({ page }) => {
    // Login first
    await page.goto(process.env.APP_URL + '/login');
    await page.fill('input[name="email"]', 'admin@example.com');
    await page.fill('input[name="password"]', 'password');
    await page.click('button[type="submit"]');

    // Navigate to organization
    await page.goto(process.env.APP_URL + '/organizations/1/dashboard');

    // Verify dashboard loads
    await expect(page.locator('h1')).toContainText('Organization Dashboard');
  });

  test('API health endpoint responds correctly', async ({ request }) => {
    const response = await request.get(process.env.APP_URL + '/api/health');

    expect(response.status()).toBe(200);

    const body = await response.json();
    expect(body.status).toBe('healthy');
    expect(body.database).toBe('healthy');
    expect(body.redis).toBe('healthy');
  });
});
```

### Performance Tests

Benchmark application performance after deployment:

**File:** `tests/Performance/DeploymentBenchmarks.php`

```php
<?php

it('responds to API requests within 200ms (95th percentile)', function () {
    $responseTimes = [];

    // Make 100 API requests
    for ($i = 0; $i < 100; $i++) {
        $start = microtime(true);
        $this->get('/api/organizations');
        $end = microtime(true);

        $responseTimes[] = ($end - $start) * 1000; // Convert to milliseconds
    }

    // Calculate 95th percentile
    sort($responseTimes);
    $p95 = $responseTimes[94];

    expect($p95)->toBeLessThan(200); // 200ms threshold
});

it('handles concurrent user requests without degradation', function () {
    // Simulate 50 concurrent users
    $requests = collect(range(1, 50))->map(function () {
        return async(fn() => $this->get('/api/organizations'));
    });

    $responses = await($requests);

    // All requests should succeed
    foreach ($responses as $response) {
        $response->assertOk();
    }
});
```

## Definition of Done

- [ ] Development environment provisioned and accessible
- [ ] Staging environment provisioned and accessible
- [ ] Production environment provisioned and accessible
- [ ] GitHub Actions workflows created (dev, staging, production)
- [ ] Quality gates workflow implemented and tested
- [ ] Reusable workflow components created
- [ ] Environment-specific configuration files created (.env.*)
- [ ] Deployment scripts implemented and tested (pre-deploy, migration, health-check, rollback, backup)
- [ ] Docker production image optimized and tested
- [ ] docker-compose.production.yml configured
- [ ] Database migration validation implemented
- [ ] Automatic rollback on migration failure tested
- [ ] Zero-downtime deployment strategy implemented (blue-green or rolling)
- [ ] Health check endpoint created at /api/health
- [ ] Automated health checks after deployment configured
- [ ] Database backup automation implemented
- [ ] S3-compatible storage configured for backups
- [ ] Secret management implemented with GitHub Secrets
- [ ] Slack/Discord notifications configured
- [ ] Manual approval workflow for production implemented
- [ ] Security scanning integrated (Trivy or equivalent)
- [ ] Performance benchmarking integrated
- [ ] Smoke tests written and passing
- [ ] Load tests written and passing
- [ ] Deployment runbook documented
- [ ] Rollback procedures documented
- [ ] Team training completed
- [ ] Successful deployment to all three environments
- [ ] PHPStan level 5 passing
- [ ] Laravel Pint formatting applied
- [ ] Code reviewed and approved
- [ ] All deployment tests passing (>90% coverage)

## Related Tasks

- **Depends on:** Task 81 (CI/CD Quality Gates) - Establishes testing and quality standards
- **Integrates with:** Task 90 (Database Migration Automation) - Migration safety mechanisms
- **Integrates with:** Task 91 (Monitoring Dashboards) - Post-deployment health monitoring
- **Enables:** All tasks 2-88 - Provides deployment infrastructure for all enterprise features
- **Benefits:** Task 2-11 (White-Label), Task 12-21 (Terraform), Task 22-31 (Monitoring), etc.


Enhance CI/CD pipeline with multi-environment deployment #196

Description

Task: Enhance CI/CD pipeline with multi-environment deployment

Description

Integration Context

Acceptance Criteria

Technical Details

File Paths

GitHub Actions Workflow Structure

Development Deployment Workflow

Staging Deployment Workflow

Production Deployment Workflow

Reusable Quality Gates Workflow

Deployment Scripts

Pre-Deployment Checks Script

Migration Validation Script

Health Check Script

Database Backup Script

Rollback Script

Production-Optimized Dockerfile

Environment Configuration Examples

Implementation Approach

Step 1: Environment Infrastructure Setup (3-4 hours)

Step 2: GitHub Actions Configuration (2-3 hours)

Step 3: Deployment Scripts (3-4 hours)

Step 4: Docker Production Optimization (2-3 hours)

Step 5: Environment Configuration (1-2 hours)

Step 6: Database Migration Safety (2-3 hours)

Step 7: Monitoring & Alerting Integration (2-3 hours)

Step 8: Testing & Validation (3-4 hours)

Step 9: Security & Compliance (1-2 hours)

Step 10: Documentation & Training (2-3 hours)

Test Strategy

Unit Tests

Integration Tests

Smoke Tests

Performance Tests

Definition of Done

Related Tasks

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions