Create examples directory, with some starter examples

docker-compose/examples/README.md

@@ -0,0 +1,46 @@
+# Docker Compose Examples
+
+## Best Practices
+
+These best practices reduce the operational cost of keeping your Sourcegraph instance up to date, and are consistent across the customers we see keeping up to date at low cost, with happy users, vs customers who are often many versions behind, with users asking, "Hey, where's the new features?"
+
+- All of your customizations should be in one docker-compose.override.yaml file, applied on top of one or more unmodified base docker-compose.yaml file(s) from this deploy-sourcegraph-docker repo
+
+- Your docker-compose.override.yaml file should be stored, maintained, and fetched from a Git repo / code version control system, with meaningful commit messages, so that future you, and future admins, can understand why changes needed to be made
+
+    - Bonus points if your override file is stored outside of your clone of this deploy-sourcegraph-docker repo, so that this deploy-sourcegraph-docker repo can be deleted / recreated / `git reset --hard` as needed, without losing your customizations
+
+- The examples in this directory use the current compose syntax, used by `docker compose`, the docker-compose-plugin. Some parts of this syntax may not be valid for the old `docker-compose` standalone binary. It is highly recommended to switch to using the new compose plugin. See Docker docs:
+
+    - Install Compose plugin
+        - https://docs.docker.com/compose/install
+
+    - Merging multiple compose files
+        - https://docs.docker.com/compose/how-tos/multiple-compose-files/merge/
+        - https://docs.docker.com/reference/compose-file/merge/#reset-value
+
+- Using multiple compose / override files
+
+    - Old method:
+
+        - Store your docker-compose.override.yaml file inside this deploy-sourcegraph-docker repo, under the /docker-compose directory
+
+        - `docker compose -f docker-compose.yaml -f executors/executor.docker-compose.yaml -f docker-compose.override.yaml up -d --remove-orphans`
+
+    - New method:
+
+        - Store your docker-compose.override.yaml file in the parent directory / beside this deploy-sourcegraph-docker repo
+
+        - Create an `.env` file, containing the below
+
+        - `docker compose up -d --remove-orphans`
+
+        - Note that any file paths (ex. mounting TLS CAs) are relative to the first compose file, so the mount path will likely start with `../../cert-chain.pem` if `cert-chain.pem` is in the parent directory / beside this deploy-sourcegraph-docker repo
+
+```env
+SOURCEGRAPH=./deploy-sourcegraph-docker/docker-compose/docker-compose.yaml
+EXECUTORS=./deploy-sourcegraph-docker/docker-compose/executors/executor.docker-compose.yaml
+OVERRIDE=./docker-compose.override.yaml
+
+COMPOSE_FILE=${SOURCEGRAPH}:${EXECUTORS}:${OVERRIDE}
+```

docker-compose/examples/aws-alb/docker-compose.override.yaml

@@ -0,0 +1,18 @@
+# Docker Compose override snippet for customers using AWS ALBs to terminate TLS,
+# and do not need TLS between the ALB and their Sourcegraph intance
+
+
+## Overrides for services defined in the base docker-compose.yaml files
+services:
+
+  sourcegraph-frontend-0:
+
+    # Add port 80, for ALB target group, in place of Caddy
+    ports:
+      - "0.0.0.0:80:3080"
+
+
+## Disable services
+
+  # Disable Caddy, because we're using an ALB for TLS termination
+  caddy: !reset {}

docker-compose/examples/aws-rds-databases/docker-compose.override.yaml

@@ -0,0 +1,142 @@
+# Docker Compose override file for customers using AWS IAM authentication to databases on RDS
+# See https://sourcegraph.com/docs/admin/external_services/postgres#usage-with-aws-rds-iam-auth
+
+
+
+## Reusable blocks of environment variables
+
+# For all containers, which connect to any of the databases
+x-database-env-var-rds-iam-auth: &database-env-var-rds-iam-auth
+
+  # Using AWS RDS IAM authentication, not a password, so instruct the psql library to refresh the token
+  PG_CONNECTION_UPDATER: EC2_ROLE_CREDENTIALS
+
+# For both migrator and frontend
+x-database-env-vars: &database-env-vars
+
+  <<: *database-env-var-rds-iam-auth
+
+  # NOTE: Must clear / override the PGDATASOURCE env vars, otherwise frontends don't start up
+  PGHOST: example-database-pgsql.cluster-example.us-west-2.rds.amazonaws.com
+  PGPORT: 5432
+  PGSSLMODE: require
+  PGDATABASE: sg
+  PGUSER: sg
+  PGPASSWORD: ""
+  PGDATASOURCE: ""
+
+  CODEINTEL_PGHOST: example-database-codeintel.cluster-example.us-west-2.rds.amazonaws.com
+  CODEINTEL_PGPORT: 5432
+  CODEINTEL_PGSSLMODE: require
+  CODEINTEL_PGDATABASE: codeintel
+  CODEINTEL_PGUSER: sg
+  CODEINTEL_PGPASSWORD: ""
+  CODEINTEL_PGDATASOURCE: ""
+
+  CODEINSIGHTS_PGHOST: example-database-codeinsights.cluster-example.us-west-2.rds.amazonaws.com
+  CODEINSIGHTS_PGPORT: 5432
+  CODEINSIGHTS_PGSSLMODE: require
+  CODEINSIGHTS_PGDATABASE: codeinsights
+  CODEINSIGHTS_PGUSER: sg
+  CODEINSIGHTS_PGPASSWORD: ""
+  CODEINSIGHTS_PGDATASOURCE: ""
+
+
+
+## Overrides for services defined in the base docker-compose.yaml files
+services:
+
+
+  gitserver-0:
+    # Add / merge the env vars, to only overwrite the ones we need, and leave the others intact
+    environment:
+      <<: *database-env-var-rds-iam-auth
+
+
+  migrator:
+
+    # !reset drops the depends_on config entirely, so that migrator will start up without the database containers
+    depends_on: !reset []
+
+    # !override drops all env vars from earlier compose files, then defines new ones, to ensure no other env vars are in the mix
+    environment: !override
+      <<: *database-env-vars
+
+    # Normal startup of migrator should only ever use the default "up" command, no additional args
+    # Overriding the command should only to be used to make specific changes, on single startups, only when needed, only when directed by Sourcegraph Customer Support Engineering
+    # If you find that your migrator needs (an) additional arg(s) to run, there's something wrong with your databases; please open a support ticket by emailing support@sourcegraph.com
+
+    # AWS RDS requires super admin permissions to install some of the extensions that Sourcegraph requires, ex. pg_stat_statements
+    # Install the needed extensions by running these queries, as an admin user, against their respective databases
+    # https://github.com/sourcegraph/artifacts/blob/main/migrations/frontend/1680296731_squashed_migrations_privileged/up.sql
+    # https://github.com/sourcegraph/artifacts/blob/main/migrations/codeintel/1679010276_squashed_migrations_privileged/up.sql
+    # https://github.com/sourcegraph/artifacts/blob/main/migrations/codeinsights/1675347548_squashed_migrations_privileged/up.sql
+    # After the extensions are successfully installed, manually mark these migrations as completed,
+    # by uncommenting these commands, one at a time, and running `docker compose up migrator` each time,
+    # then comment the last one out, and run the normal `docker compose up -d --remove-orphans` command, for a normal startup
+    # command: ["add-log", "-db=frontend", "-version=1680296731"]
+    # command: ["add-log", "-db=codeintel", "-version=1679010276"]
+    # command: ["add-log", "-db=codeinsights", "-version=1675347548"]
+
+    # If you do fall a few versions behind, here's the command to run a multi-version upgrade
+    # See https://sourcegraph.com/docs/admin/deploy/docker-compose/upgrade#multi-version-upgrades
+    # command: ["upgrade", "--from=v6.4.0", "--to=v6.9.1277"]
+
+    # Do not skip the drift check, or use other args, without instruction from Sourcegraph Customer Support Engineering
+    # If migrator fails to start up due to drift detected, or other issues, please open a support ticket by emailing support@sourcegraph.com
+
+
+  precise-code-intel-worker:
+    # Add / merge the env vars, to only overwrite the ones we need, and leave the others intact
+    environment:
+      <<: *database-env-var-rds-iam-auth
+
+
+  searcher-0:
+    # Add / merge the env vars, to only overwrite the ones we need, and leave the others intact
+    environment:
+      <<: *database-env-var-rds-iam-auth
+
+
+  sourcegraph-frontend-0:
+    # Add / merge the env vars, to only overwrite the ones we need, and leave the others intact
+    environment:
+      <<: *database-env-vars
+
+
+  sourcegraph-frontend-internal:
+    # !override drops all depends_on from earlier compose files, then defines new ones,
+    # to ensure it doesn't wait for the database containers, but also doesn't skip the dependency on migrator
+    depends_on: !override
+      migrator:
+        condition: service_completed_successfully
+    # Add / merge the env vars, to only overwrite the ones we need, and leave the others intact
+    environment:
+      <<: *database-env-vars
+
+
+  syntactic-code-intel-worker:
+    # Add / merge the env vars, to only overwrite the ones we need, and leave the others intact
+    environment:
+      <<: *database-env-var-rds-iam-auth
+
+
+  worker:
+    # Add / merge the env vars, to only overwrite the ones we need, and leave the others intact
+    environment:
+      <<: *database-env-var-rds-iam-auth
+      # Add additional env vars, as needed
+      # TELEMETRY_GATEWAY_EXPORTER_USERMETADATA_EXPORT_ENABLED: true
+      # TELEMETRY_GATEWAY_EXPORTER_USERMETADATA_EXPORT_ORGANIZATIONS: true
+      # TELEMETRY_GATEWAY_EXPORTER_USERMETADATA_EXPORT_USERS: true
+
+
+## Disable services
+
+  # Drop built-in database services and volumes from the base compose file
+  pgsql: !reset {}
+  pgsql-exporter: !reset {}
+  codeintel-db: !reset {}
+  codeintel-db-exporter: !reset {}
+  codeinsights-db: !reset {}
+  codeinsights-db-exporter: !reset {}

docker-compose/examples/migrator-operations/docker-compose.override.yaml

@@ -0,0 +1,40 @@
+## Overrides for services defined in the base docker-compose.yaml files
+services:
+
+  migrator:
+
+    ################################################################################
+    # Normal startup of migrator should only ever use the default "up" command, no additional args
+    # Overriding the command should only to be used to make specific changes, on single startups, only when needed, only when directed by Sourcegraph Customer Support Engineering
+    # If you find that your migrator needs (an) additional arg(s) to run, there's something wrong with your databases; please open a support ticket by emailing support@sourcegraph.com
+    command: ["up"]
+    ################################################################################
+
+
+    ################################################################################
+    # Initializing new databases
+
+    # AWS RDS, and some other DMBSs require super admin permissions to install some of the extensions that Sourcegraph requires, ex. pg_stat_statements
+
+    # Install the needed extensions by running these queries, as an admin user, against their respective databases
+    # https://github.com/sourcegraph/artifacts/blob/main/migrations/frontend/1680296731_squashed_migrations_privileged/up.sql
+    # https://github.com/sourcegraph/artifacts/blob/main/migrations/codeintel/1679010276_squashed_migrations_privileged/up.sql
+    # https://github.com/sourcegraph/artifacts/blob/main/migrations/codeinsights/1675347548_squashed_migrations_privileged/up.sql
+
+    # After the extensions are successfully installed, manually mark these migrations as completed,
+    # by uncommenting these commands, one at a time, and running `docker compose up migrator` each time,
+    # then comment the last one out, and run the normal `docker compose up -d --remove-orphans` command, for a normal startup
+    # command: ["add-log", "-db=frontend", "-version=1680296731"]
+    # command: ["add-log", "-db=codeintel", "-version=1679010276"]
+    # command: ["add-log", "-db=codeinsights", "-version=1675347548"]
+    ################################################################################
+
+
+    ################################################################################
+    # Multi-version upgrades
+    # See https://sourcegraph.com/docs/admin/deploy/docker-compose/upgrade#multi-version-upgrades
+    # command: ["upgrade", "--from=v6.4.0", "--to=v6.9.1277"]
+
+    # Do not skip the drift check, or use other args, without instruction from Sourcegraph Customer Support Engineering
+    # If migrator fails to start up due to drift detected, or other issues, please open a support ticket by emailing support@sourcegraph.com
+    ################################################################################