From bbbcfb8d5d5cc42d9ab96dba2c0332c9613e411e Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Fri, 23 Jan 2026 15:49:26 +0000
Subject: [PATCH 01/12] Add token rotation runbook and refactor
 BWS_ACCESS_TOKEN naming

- Create comprehensive token rotation runbook documenting all CI/CD
  tokens and secrets with rotation procedures
- Refactor PR workflow to use BWS_ACCESS_TOKEN_DEV for repository-level
  secret (matches ADMIN_IP_DEV naming pattern)
- Environment-scoped BWS_ACCESS_TOKEN remains for deploy workflows
---
 .github/workflows/pr-tofu-plan-develop.yml |   2 +-
 docs/token-rotation-runbook.md             | 599 +++++++++++++++++++++
 2 files changed, 600 insertions(+), 1 deletion(-)
 create mode 100644 docs/token-rotation-runbook.md

diff --git a/.github/workflows/pr-tofu-plan-develop.yml b/.github/workflows/pr-tofu-plan-develop.yml
index 9c687aa..ae7d3c1 100644
--- a/.github/workflows/pr-tofu-plan-develop.yml
+++ b/.github/workflows/pr-tofu-plan-develop.yml
@@ -49,7 +49,7 @@ jobs:
 
       - name: Retrieve secrets via infra-shell.sh (CI mode)
         env:
-          BWS_ACCESS_TOKEN: ${{ secrets.BWS_ACCESS_TOKEN }}
+          BWS_ACCESS_TOKEN: ${{ secrets.BWS_ACCESS_TOKEN_DEV }}
           # GitHub repository variable for R2 bucket name
           BOOTSTRAP_R2_BUCKET_DEV: ${{ vars.BOOTSTRAP_R2_BUCKET_DEV }}
           # GitHub repository secret for workstation IP (used for SSH firewall rules)
diff --git a/docs/token-rotation-runbook.md b/docs/token-rotation-runbook.md
new file mode 100644
index 0000000..61d6a65
--- /dev/null
+++ b/docs/token-rotation-runbook.md
@@ -0,0 +1,599 @@
+# Token Rotation Runbook
+
+This document provides step-by-step procedures for rotating all tokens and secrets used in the ghost-stack infrastructure. Regular rotation is critical for maintaining security hygiene.
+
+---
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Token Inventory](#token-inventory)
+3. [GitHub Tokens](#github-tokens)
+4. [Bitwarden Secrets Manager](#bitwarden-secrets-manager)
+5. [Cloudflare Tokens](#cloudflare-tokens)
+6. [R2 Storage Credentials](#r2-storage-credentials)
+7. [Vultr API Key](#vultr-api-key)
+8. [Tailscale API Key](#tailscale-api-key)
+9. [PagerDuty Credentials](#pagerduty-credentials)
+10. [Grafana Cloud Credentials](#grafana-cloud-credentials)
+11. [Linear API Token](#linear-api-token)
+12. [Verification Procedures](#verification-procedures)
+
+---
+
+## Overview
+
+### Storage Locations
+
+Secrets in this project are stored in two primary locations:
+
+| Location | Purpose | Access Method |
+|----------|---------|---------------|
+| **Bitwarden Secrets Manager** | Runtime secrets for OpenTofu and scripts | `bws` CLI via `infra-shell.sh` |
+| **GitHub Secrets** | CI/CD workflow secrets | GitHub Actions environment variables |
+
+### GitHub Secrets Scoping
+
+GitHub secrets are scoped at two levels:
+
+| Scope | Usage | Naming Convention |
+|-------|-------|-------------------|
+| **Repository-level** | PR workflows (cannot access environment secrets) | `SECRET_NAME_DEV` suffix |
+| **Environment-scoped** | Deploy workflows (`environment: dev`) | `SECRET_NAME` (no suffix) |
+
+**Important:** When a secret is environment-scoped, you must update it in the GitHub environment settings (Settings → Environments → dev), not in the repository secrets.
+
+---
+
+## Token Inventory
+
+### Quick Reference Table
+
+| Token | Source | Bitwarden ID | GitHub Secret | Env Scope | Expiration |
+|-------|--------|--------------|---------------|-----------|------------|
+| GHCR RW Token | GitHub PAT | N/A | `GHCR_TOKEN` | Repository | Configurable |
+| BWS Access Token | Bitwarden | N/A | `BWS_ACCESS_TOKEN` | Environment (dev) | Never* |
+| BWS Access Token | Bitwarden | N/A | `BWS_ACCESS_TOKEN_DEV` | Repository | Never* |
+| Claude MCP Token | GitHub PAT | N/A | N/A (local) | N/A | Configurable |
+| Cloudflare API Token | Cloudflare | `59624245-...` | N/A | N/A | Configurable |
+| Cloudflare Token Creator | Cloudflare | N/A | N/A | N/A | 30 days recommended |
+| Cloudflare Bootstrap Token | Cloudflare | N/A | N/A | N/A | 30 days recommended |
+| R2 Access Key ID | Cloudflare R2 | `9dfdf110-...` | N/A | N/A | Never |
+| R2 Secret Access Key | Cloudflare R2 | `f5d9794d-...` | N/A | N/A | Never |
+| R2 Bootstrap Access Key | Cloudflare R2 | N/A | N/A | N/A | Never |
+| R2 Bootstrap Secret Key | Cloudflare R2 | N/A | N/A | N/A | Never |
+| Vultr API Key | Vultr | `d68b6562-...` | N/A | N/A | Never |
+| Tailscale API Key | Tailscale | `34b620b7-...` | N/A | N/A | 90 days default |
+| PagerDuty Client ID | PagerDuty | `7d51661b-...` | N/A | N/A | Never |
+| PagerDuty Client Secret | PagerDuty | `b15575c0-...` | N/A | N/A | Never |
+| PagerDuty User Token | PagerDuty | `02805292-...` | N/A | N/A | Never |
+| Grafana Cloud Token | Grafana | `bfc8dd06-...` | N/A | N/A | Never |
+| Grafana Cloud SA Token | Grafana | `3ebc4398-...` | N/A | N/A | Never |
+| Linear API Token | Linear | N/A | N/A (local) | N/A | Never |
+| Admin IP | N/A | N/A | `ADMIN_IP` / `ADMIN_IP_DEV` | Both | N/A |
+| Cloudflare Zone ID | N/A | N/A | `CLOUDFLARE_ZONE_ID` / `CLOUDFLARE_ZONE_ID_DEV` | Both | N/A |
+| Health Check Token | N/A | N/A | `HEALTH_CHECK_TOKEN` | Environment (dev) | N/A |
+
+*Bitwarden machine account tokens do not expire but should be rotated periodically.
+
+---
+
+## GitHub Tokens
+
+### GHCR Read/Write Token (`GHCR_TOKEN`)
+
+**Purpose:** Authenticate to GitHub Container Registry to pull the `ghost-stack-shell` image in CI/CD workflows.
+
+**Scope:** Repository-level (used by both PR and deploy workflows)
+
+**Expiration:** Configurable (recommend 90 days)
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Go to GitHub → Settings → Developer settings → Personal access tokens → Tokens (classic)
+   - Click "Generate new token (classic)"
+   - Name: `ghost-stack-ghcr-rw`
+   - Expiration: 90 days (or your preferred period)
+   - Scopes: `write:packages`, `read:packages`
+   - Click "Generate token"
+   - Copy the token immediately
+
+2. **Update GitHub Secret:**
+   - Go to `github.com/noahwhite/ghost-stack` → Settings → Secrets and variables → Actions
+   - Find `GHCR_TOKEN` under Repository secrets
+   - Click "Update"
+   - Paste the new token
+   - Click "Update secret"
+
+3. **Verify:**
+   - Trigger a workflow that uses GHCR (e.g., create a draft PR)
+   - Confirm the "Log in to GHCR" step succeeds
+
+---
+
+### Claude GitHub MCP Access Token
+
+**Purpose:** Allows Claude Code to interact with GitHub via MCP (Model Context Protocol) for issue management, PR creation, etc.
+
+**Scope:** Local development only (not stored in GitHub)
+
+**Expiration:** Configurable (recommend 90 days)
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Go to GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens
+   - Click "Generate new token"
+   - Name: `claude-mcp-access`
+   - Expiration: 90 days
+   - Repository access: Select repositories → choose `ghost-stack`, `alloy-sysext-build`
+   - Permissions:
+     - Contents: Read and write
+     - Issues: Read and write
+     - Pull requests: Read and write
+     - Metadata: Read-only
+   - Click "Generate token"
+
+2. **Update local configuration:**
+   - Update your Claude Code MCP configuration with the new token
+   - Location varies by setup (typically `~/.config/claude/mcp.json` or similar)
+
+3. **Verify:**
+   - Use Claude Code to list issues or create a test comment
+
+---
+
+## Bitwarden Secrets Manager
+
+### BWS Access Token (`BWS_ACCESS_TOKEN`)
+
+**Purpose:** Authenticate to Bitwarden Secrets Manager to retrieve runtime secrets in CI/CD.
+
+**Scope:**
+- **Environment-scoped (`BWS_ACCESS_TOKEN`):** Used by deploy workflows with `environment: dev`
+- **Repository-level (`BWS_ACCESS_TOKEN_DEV`):** Used by PR workflows (matches `ADMIN_IP_DEV` pattern)
+
+This naming convention provides clear environment isolation and aligns with other secrets like `ADMIN_IP_DEV` and `CLOUDFLARE_ZONE_ID_DEV`.
+
+**Expiration:** Machine account tokens do not expire, but rotation is recommended every 6-12 months.
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Log into Bitwarden web vault
+   - Go to Organizations → Machine accounts
+   - Select the relevant machine account (e.g., `ghost-stack-dev`)
+   - Go to Access tokens tab
+   - Click "Create access token"
+   - Name: Include date (e.g., `ci-2025-01`)
+   - Copy the token immediately (shown only once)
+
+2. **Update GitHub Secrets:**
+
+   For **environment-scoped** (deploy workflows):
+   - Go to `github.com/noahwhite/ghost-stack` → Settings → Environments → dev
+   - Find `BWS_ACCESS_TOKEN`
+   - Click pencil icon → Update → paste new token → Update secret
+
+   For **repository-level** (PR workflows):
+   - Go to Settings → Secrets and variables → Actions
+   - Find `BWS_ACCESS_TOKEN_DEV` under Repository secrets
+   - Click "Update" → paste new token → Update secret
+
+3. **Revoke old token:**
+   - In Bitwarden, delete the old access token from the machine account
+
+4. **Verify:**
+   - Trigger a deploy workflow
+   - Confirm secrets retrieval succeeds in the logs
+
+---
+
+## Cloudflare Tokens
+
+### Cloudflare API Token (OpenTofu)
+
+**Purpose:** Manage Cloudflare resources (DNS, Page Rules) via OpenTofu.
+
+**Bitwarden Secret ID:** `59624245-6a0c-4fde-9d6d-b39c014882a6`
+
+**Expiration:** Configurable at creation
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Log into Cloudflare dashboard (dev account)
+   - Go to My Profile → API Tokens → Create Token
+   - Use "Edit zone DNS" template or custom:
+     - Zone: DNS: Edit
+     - Zone: Zone: Read
+     - Zone Resources: Include specific zone or all zones
+   - Set IP restrictions if desired
+   - Set TTL (recommend 90 days)
+   - Create token and copy immediately
+
+2. **Update Bitwarden:**
+   - Log into Bitwarden web vault
+   - Find secret with ID `59624245-6a0c-4fde-9d6d-b39c014882a6`
+   - Update the value with the new token
+   - Save
+
+3. **Revoke old token:**
+   - In Cloudflare, go to My Profile → API Tokens
+   - Find the old token and click "Revoke"
+
+4. **Verify:**
+   - Run `./opentofu/scripts/tofu.sh dev plan`
+   - Confirm no authentication errors
+
+---
+
+### Cloudflare Token Creator (`dev-token-creator`)
+
+**Purpose:** Create other scoped Cloudflare API tokens programmatically.
+
+**Storage:** Bitwarden Secrets Manager
+
+**Expiration:** 30 days recommended
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Log into Cloudflare dashboard (dev account)
+   - Go to My Profile → API Tokens → Create Token
+   - Select template: "Create Additional Tokens"
+   - Permissions: User, API Tokens, Edit
+   - Set IP restrictions to your admin IP
+   - Set TTL: 30 days
+   - Create and copy token
+
+2. **Update Bitwarden:**
+   - Update the `dev-token-creator` secret in Bitwarden
+
+3. **Revoke old token:**
+   - In Cloudflare, revoke the previous token creator
+
+---
+
+### Cloudflare Bootstrap Token
+
+**Purpose:** Provision R2 bucket and DNS zone during initial bootstrap.
+
+**Storage:** Bitwarden Secrets Manager
+
+**Expiration:** 30 days recommended
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Use the token creator script:
+     ```bash
+     ./opentofu/bootstrap/scripts/generate-bootstrap-token.sh
+     ```
+   - Or manually create in Cloudflare with permissions:
+     - Zone: Edit, Read
+     - DNS: Edit
+     - R2 Storage Buckets: Edit
+
+2. **Update Bitwarden:**
+   - Update the `bootstrap-dev-token` secret in Bitwarden
+
+---
+
+## R2 Storage Credentials
+
+### R2 Access Key ID & Secret Access Key
+
+**Purpose:** Access R2 buckets for OpenTofu state storage and sysext image storage.
+
+**Bitwarden Secret IDs:**
+- Access Key ID: `9dfdf110-5a84-48c3-ad7e-b39b002afd6b`
+- Secret Access Key: `f5d9794d-fd45-4dcb-9994-b39b002b5056`
+
+**Expiration:** Never (but rotate periodically)
+
+#### Rotation Steps
+
+1. **Generate new credentials:**
+   - Log into Cloudflare dashboard
+   - Go to R2 → Overview → Manage R2 API Tokens
+   - Click "Create API token"
+   - Name: `ghost-stack-r2-YYYY-MM`
+   - Permissions: Object Read & Write
+   - Specify bucket(s): `ghost-stack-dev-state`, `ghost-dev-sysext-images`
+   - TTL: None (or set expiration)
+   - Create and copy both Access Key ID and Secret Access Key
+
+2. **Update Bitwarden:**
+   - Update secret `9dfdf110-5a84-48c3-ad7e-b39b002afd6b` with new Access Key ID
+   - Update secret `f5d9794d-fd45-4dcb-9994-b39b002b5056` with new Secret Access Key
+
+3. **Revoke old credentials:**
+   - In Cloudflare R2, delete the old API token
+
+4. **Verify:**
+   - Run `./opentofu/scripts/tofu.sh dev plan`
+   - Confirm state can be read/written
+
+---
+
+### R2 Bootstrap Credentials
+
+**Purpose:** Bootstrap R2 bucket creation (used only during initial setup).
+
+**Storage:** Bitwarden Secrets Manager
+
+**Rotation:** Only needed if re-bootstrapping infrastructure
+
+#### Rotation Steps
+
+1. **Generate new credentials:**
+   - Log into Cloudflare dashboard
+   - Go to R2 → Overview → Manage R2 API Tokens
+   - Create new token with R2 bucket creation permissions
+   - Copy Access Key ID and Secret Access Key
+
+2. **Update Bitwarden:**
+   - Update the bootstrap R2 access key and secret key secrets in Bitwarden
+
+---
+
+## Vultr API Key
+
+**Purpose:** Manage Vultr compute instances, firewalls, and block storage.
+
+**Bitwarden Secret ID:** `d68b6562-0d9e-424c-b2c5-b39c013ae34d`
+
+**Expiration:** Never
+
+#### Rotation Steps
+
+1. **Generate new key:**
+   - Log into Vultr (dev account)
+   - Go to Account → API
+   - Click "Enable API" if not already enabled
+   - Copy the API key (or regenerate if rotating)
+
+   **Note:** Vultr only supports one API key per account. Regenerating creates a new key and invalidates the old one immediately.
+
+2. **Update Bitwarden:**
+   - Update secret `d68b6562-0d9e-424c-b2c5-b39c013ae34d` with new key
+
+3. **Verify:**
+   - Run `./opentofu/scripts/tofu.sh dev plan`
+   - Confirm Vultr resources are accessible
+
+---
+
+## Tailscale API Key
+
+**Purpose:** Register/deregister Tailscale devices via OpenTofu.
+
+**Bitwarden Secret IDs:**
+- API Key: `34b620b7-edf6-4d06-9792-b39b00317467`
+- Tailnet: `a8f07ce5-ed4d-42bb-b012-b39b00311d41`
+
+**Expiration:** 90 days by default
+
+#### Rotation Steps
+
+1. **Generate new key:**
+   - Log into Tailscale admin console
+   - Go to Settings → Keys
+   - Click "Generate API key"
+   - Description: `ghost-stack-tofu-YYYY-MM`
+   - Expiry: 90 days
+   - Copy the key
+
+2. **Update Bitwarden:**
+   - Update secret `34b620b7-edf6-4d06-9792-b39b00317467` with new key
+
+3. **Revoke old key:**
+   - In Tailscale, delete the old API key
+
+4. **Verify:**
+   - Run `./opentofu/scripts/tofu.sh dev plan`
+   - Confirm Tailscale provider initializes
+
+---
+
+## PagerDuty Credentials
+
+### PagerDuty OAuth Credentials
+
+**Purpose:** Configure PagerDuty integrations via OpenTofu.
+
+**Bitwarden Secret IDs:**
+- Subdomain: `8ee84397-e563-4278-9a3f-b39c013f7575`
+- Client ID: `7d51661b-736a-43ff-b01f-b39c013fe49b`
+- Client Secret: `b15575c0-0d28-459d-b92d-b39c01403a38`
+
+**Expiration:** Never
+
+#### Rotation Steps
+
+1. **Regenerate OAuth credentials:**
+   - Log into PagerDuty
+   - Go to Integrations → Developer Mode → My Apps
+   - Find your OAuth app
+   - Regenerate client secret (this invalidates the old one)
+
+2. **Update Bitwarden:**
+   - Update secret `b15575c0-0d28-459d-b92d-b39c01403a38` with new Client Secret
+
+---
+
+### PagerDuty User API Token
+
+**Purpose:** User-level API access for PagerDuty operations.
+
+**Bitwarden Secret ID:** `02805292-4311-4290-9b6e-b39c01554ae6`
+
+**Expiration:** Never
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Log into PagerDuty
+   - Go to My Profile → User Settings → Create API User Token
+   - Description: `ghost-stack-tofu-YYYY-MM`
+   - Copy the token
+
+2. **Update Bitwarden:**
+   - Update secret `02805292-4311-4290-9b6e-b39c01554ae6` with new token
+
+3. **Revoke old token:**
+   - Delete the old API user token in PagerDuty
+
+---
+
+## Grafana Cloud Credentials
+
+### Grafana Cloud Access Token
+
+**Purpose:** Configure Grafana Cloud observability via OpenTofu.
+
+**Bitwarden Secret ID:** `bfc8dd06-bd97-499a-98f8-b3a101570606`
+
+**Expiration:** Never (but rotation recommended)
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Log into Grafana Cloud
+   - Go to My Account → Access Policies
+   - Create new access policy or token with required permissions
+   - Copy the token
+
+2. **Update Bitwarden:**
+   - Update secret `bfc8dd06-bd97-499a-98f8-b3a101570606` with new token
+
+3. **Revoke old token:**
+   - Delete the old access policy/token in Grafana Cloud
+
+---
+
+### Grafana Cloud Terraform Service Account Token
+
+**Purpose:** Service account for Grafana Cloud Terraform provider (doc project).
+
+**Bitwarden Secret ID:** `3ebc4398-f4fa-448c-b2c1-b3a6006c063d`
+
+**Expiration:** Configurable
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Log into Grafana Cloud
+   - Go to Administration → Service accounts
+   - Find the relevant service account
+   - Create new token
+   - Copy the token
+
+2. **Update Bitwarden:**
+   - Update secret `3ebc4398-f4fa-448c-b2c1-b3a6006c063d` with new token
+
+3. **Delete old token:**
+   - Remove the old token from the service account
+
+---
+
+## Linear API Token
+
+**Purpose:** Claude Code integration with Linear for issue tracking.
+
+**Storage:** Local Claude Code MCP configuration
+
+**Expiration:** Never
+
+#### Rotation Steps
+
+1. **Generate new token:**
+   - Log into Linear
+   - Go to Settings → API → Personal API keys
+   - Click "Create key"
+   - Label: `claude-code-YYYY-MM`
+   - Copy the token
+
+2. **Update local configuration:**
+   - Update your Claude Code MCP configuration with the new token
+
+3. **Revoke old token:**
+   - Delete the old API key in Linear
+
+---
+
+## Verification Procedures
+
+After rotating any token, perform the following verifications:
+
+### CI/CD Workflow Verification
+
+1. **PR Workflow:**
+   - Create a draft PR with a minor change
+   - Verify all workflow steps pass:
+     - Log in to GHCR
+     - Retrieve secrets from Bitwarden
+     - OpenTofu plan executes successfully
+
+2. **Deploy Workflow:**
+   - Trigger a manual workflow run or merge a PR
+   - Verify deployment completes successfully
+
+### Local Development Verification
+
+1. **OpenTofu:**
+   ```bash
+   source docker/scripts/infra-shell.sh
+   ./opentofu/scripts/tofu.sh dev plan
+   ```
+   Confirm no authentication errors.
+
+2. **Bitwarden:**
+   ```bash
+   bws secret list
+   ```
+   Confirm secrets are accessible.
+
+### Service-Specific Verification
+
+| Service | Verification Command/Action |
+|---------|---------------------------|
+| Cloudflare | `curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" -H "Authorization: Bearer $TOKEN"` |
+| Vultr | `curl -H "Authorization: Bearer $VULTR_API_KEY" https://api.vultr.com/v2/account` |
+| Tailscale | Check admin console for API key status |
+| PagerDuty | OpenTofu plan with PagerDuty resources |
+| Grafana | OpenTofu plan with Grafana resources |
+
+---
+
+## Rotation Schedule Recommendations
+
+| Token | Recommended Rotation | Priority |
+|-------|---------------------|----------|
+| GHCR Token | Every 90 days | High |
+| Cloudflare API Tokens | Every 90 days | High |
+| Tailscale API Key | Before 90-day expiry | High |
+| BWS Access Tokens | Every 6-12 months | Medium |
+| R2 Credentials | Every 6-12 months | Medium |
+| Vultr API Key | Annually | Medium |
+| PagerDuty Tokens | Annually | Low |
+| Grafana Tokens | Annually | Low |
+| Linear API Token | Annually | Low |
+
+---
+
+## Emergency Rotation
+
+If a token is suspected to be compromised:
+
+1. **Immediately revoke** the compromised token at its source
+2. **Generate a new token** following the steps above
+3. **Update all storage locations** (Bitwarden, GitHub Secrets)
+4. **Audit logs** for unauthorized access
+5. **Document the incident** and review access patterns
+
+---
+
+_This document lives at `docs/token-rotation-runbook.md` in the repository._

From de147bc47c6812b124e2b77f5d24e9a072729cc3 Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sat, 24 Jan 2026 01:14:22 +0000
Subject: [PATCH 02/12] Use environment-scoped BWS_ACCESS_TOKEN for PR workflow

- Add `environment: dev` to PR workflow job
- Use environment-scoped BWS_ACCESS_TOKEN (no _DEV suffix needed)
- Update runbook to reflect simplified secret scoping
---
 .github/workflows/pr-tofu-plan-develop.yml |  3 ++-
 docs/token-rotation-runbook.md             | 16 +++-------------
 2 files changed, 5 insertions(+), 14 deletions(-)

diff --git a/.github/workflows/pr-tofu-plan-develop.yml b/.github/workflows/pr-tofu-plan-develop.yml
index ae7d3c1..ff40fd3 100644
--- a/.github/workflows/pr-tofu-plan-develop.yml
+++ b/.github/workflows/pr-tofu-plan-develop.yml
@@ -14,6 +14,7 @@ permissions:
 jobs:
   tofu-plan:
     runs-on: ubuntu-latest
+    environment: dev
 
     steps:
       - name: Install Bitwarden Secrets Manager CLI (bws)
@@ -49,7 +50,7 @@ jobs:
 
       - name: Retrieve secrets via infra-shell.sh (CI mode)
         env:
-          BWS_ACCESS_TOKEN: ${{ secrets.BWS_ACCESS_TOKEN_DEV }}
+          BWS_ACCESS_TOKEN: ${{ secrets.BWS_ACCESS_TOKEN }}
           # GitHub repository variable for R2 bucket name
           BOOTSTRAP_R2_BUCKET_DEV: ${{ vars.BOOTSTRAP_R2_BUCKET_DEV }}
           # GitHub repository secret for workstation IP (used for SSH firewall rules)
diff --git a/docs/token-rotation-runbook.md b/docs/token-rotation-runbook.md
index 61d6a65..2e2f635 100644
--- a/docs/token-rotation-runbook.md
+++ b/docs/token-rotation-runbook.md
@@ -53,7 +53,6 @@ GitHub secrets are scoped at two levels:
 |-------|--------|--------------|---------------|-----------|------------|
 | GHCR RW Token | GitHub PAT | N/A | `GHCR_TOKEN` | Repository | Configurable |
 | BWS Access Token | Bitwarden | N/A | `BWS_ACCESS_TOKEN` | Environment (dev) | Never* |
-| BWS Access Token | Bitwarden | N/A | `BWS_ACCESS_TOKEN_DEV` | Repository | Never* |
 | Claude MCP Token | GitHub PAT | N/A | N/A (local) | N/A | Configurable |
 | Cloudflare API Token | Cloudflare | `59624245-...` | N/A | N/A | Configurable |
 | Cloudflare Token Creator | Cloudflare | N/A | N/A | N/A | 30 days recommended |
@@ -150,11 +149,9 @@ GitHub secrets are scoped at two levels:
 
 **Purpose:** Authenticate to Bitwarden Secrets Manager to retrieve runtime secrets in CI/CD.
 
-**Scope:**
-- **Environment-scoped (`BWS_ACCESS_TOKEN`):** Used by deploy workflows with `environment: dev`
-- **Repository-level (`BWS_ACCESS_TOKEN_DEV`):** Used by PR workflows (matches `ADMIN_IP_DEV` pattern)
+**Scope:** Environment-scoped (`environment: dev`)
 
-This naming convention provides clear environment isolation and aligns with other secrets like `ADMIN_IP_DEV` and `CLOUDFLARE_ZONE_ID_DEV`.
+Both PR and deploy workflows use `environment: dev`, so a single environment-scoped secret is sufficient.
 
 **Expiration:** Machine account tokens do not expire, but rotation is recommended every 6-12 months.
 
@@ -169,18 +166,11 @@ This naming convention provides clear environment isolation and aligns with othe
    - Name: Include date (e.g., `ci-2025-01`)
    - Copy the token immediately (shown only once)
 
-2. **Update GitHub Secrets:**
-
-   For **environment-scoped** (deploy workflows):
+2. **Update GitHub Secret:**
    - Go to `github.com/noahwhite/ghost-stack` → Settings → Environments → dev
    - Find `BWS_ACCESS_TOKEN`
    - Click pencil icon → Update → paste new token → Update secret
 
-   For **repository-level** (PR workflows):
-   - Go to Settings → Secrets and variables → Actions
-   - Find `BWS_ACCESS_TOKEN_DEV` under Repository secrets
-   - Click "Update" → paste new token → Update secret
-
 3. **Revoke old token:**
    - In Bitwarden, delete the old access token from the machine account
 

From db8c8b532fd8e43b5b351b76c31efbcfa32f7276 Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sat, 24 Jan 2026 01:17:17 +0000
Subject: [PATCH 03/12] Use environment-scoped ADMIN_IP and CLOUDFLARE_ZONE_ID

Remove _DEV suffix pattern - now that PR workflow uses environment: dev,
it can access environment-scoped secrets directly.

Repository-level copies (ADMIN_IP_DEV, CLOUDFLARE_ZONE_ID_DEV) can be deleted.
---
 .github/workflows/pr-tofu-plan-develop.yml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/.github/workflows/pr-tofu-plan-develop.yml b/.github/workflows/pr-tofu-plan-develop.yml
index ff40fd3..43b5eef 100644
--- a/.github/workflows/pr-tofu-plan-develop.yml
+++ b/.github/workflows/pr-tofu-plan-develop.yml
@@ -53,10 +53,10 @@ jobs:
           BWS_ACCESS_TOKEN: ${{ secrets.BWS_ACCESS_TOKEN }}
           # GitHub repository variable for R2 bucket name
           BOOTSTRAP_R2_BUCKET_DEV: ${{ vars.BOOTSTRAP_R2_BUCKET_DEV }}
-          # GitHub repository secret for workstation IP (used for SSH firewall rules)
-          ADMIN_IP_DEV: ${{ secrets.ADMIN_IP_DEV }}
-          # GitHub repository secret for Cloudflare Zone ID (from bootstrap outputs)
-          CLOUDFLARE_ZONE_ID_DEV: ${{ secrets.CLOUDFLARE_ZONE_ID_DEV }}
+          # GitHub environment secret for workstation IP (used for SSH firewall rules)
+          ADMIN_IP: ${{ secrets.ADMIN_IP }}
+          # GitHub environment secret for Cloudflare Zone ID (from bootstrap outputs)
+          CLOUDFLARE_ZONE_ID: ${{ secrets.CLOUDFLARE_ZONE_ID }}
         run: |
           ./docker/scripts/infra-shell.sh --ci --secrets-only --export-github-env
 

From f72de6a3bd1495aae9fadd14e8f94241251eab5a Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sat, 24 Jan 2026 01:17:28 +0000
Subject: [PATCH 04/12] Update runbook: ADMIN_IP and CLOUDFLARE_ZONE_ID are
 environment-scoped

---
 docs/token-rotation-runbook.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/token-rotation-runbook.md b/docs/token-rotation-runbook.md
index 2e2f635..d6d14a7 100644
--- a/docs/token-rotation-runbook.md
+++ b/docs/token-rotation-runbook.md
@@ -69,8 +69,8 @@ GitHub secrets are scoped at two levels:
 | Grafana Cloud Token | Grafana | `bfc8dd06-...` | N/A | N/A | Never |
 | Grafana Cloud SA Token | Grafana | `3ebc4398-...` | N/A | N/A | Never |
 | Linear API Token | Linear | N/A | N/A (local) | N/A | Never |
-| Admin IP | N/A | N/A | `ADMIN_IP` / `ADMIN_IP_DEV` | Both | N/A |
-| Cloudflare Zone ID | N/A | N/A | `CLOUDFLARE_ZONE_ID` / `CLOUDFLARE_ZONE_ID_DEV` | Both | N/A |
+| Admin IP | N/A | N/A | `ADMIN_IP` | Environment (dev) | N/A |
+| Cloudflare Zone ID | N/A | N/A | `CLOUDFLARE_ZONE_ID` | Environment (dev) | N/A |
 | Health Check Token | N/A | N/A | `HEALTH_CHECK_TOKEN` | Environment (dev) | N/A |
 
 *Bitwarden machine account tokens do not expire but should be rotated periodically.

From 2e388d7303ce40f495fd49059b326ff469d5bdf5 Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sat, 24 Jan 2026 01:30:21 +0000
Subject: [PATCH 05/12] Update CLAUDE.md: no Claude attribution, fix secrets
 docs

- Add rule to never add "Generated with Claude Code" to PRs/commits
- Update secrets management section to reflect environment-scoped pattern
- Reference token-rotation-runbook.md for complete token inventory
---
 CLAUDE.md | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index be6c0ad..c76c212 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -9,6 +9,7 @@ You are a staff-level infrastructure and application engineer/architect. Provide
 - All commit messages should be clear and descriptive
 - All PR comments must be formatted in markdown
 - Use todo lists to track multi-step tasks
+- **Never add "Generated with Claude Code" or similar attribution lines to PRs or commits**
 
 ## Testing Requirements
 
@@ -130,9 +131,10 @@ docs/                   # Documentation
 ## Important Patterns
 
 ### Secrets Management
-- **Environment-scoped secrets**: Used in deploy workflows (e.g., `ADMIN_IP`, `CLOUDFLARE_ZONE_ID`)
-- **Repository-level secrets with `_DEV` suffix**: Used in PR workflows (can't access environment secrets)
+- **Environment-scoped secrets**: Used by both PR and deploy workflows (e.g., `BWS_ACCESS_TOKEN`, `ADMIN_IP`, `CLOUDFLARE_ZONE_ID`)
+- **Repository-level secrets**: Only `GHCR_TOKEN` remains at repository level (for workflows without environment)
 - **Bitwarden Secrets Manager**: Retrieves secrets at runtime via `infra-shell.sh`
+- See `docs/token-rotation-runbook.md` for complete token inventory and rotation procedures
 
 ### OpenTofu Wrapper Script
 Use `./opentofu/scripts/tofu.sh` instead of `tofu` directly:

From 03b31cde89edcf45182c1686ac0d578eac57ca31 Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sat, 24 Jan 2026 01:41:22 +0000
Subject: [PATCH 06/12] Add Co-Authored-By, PR workflow, and assignee rules to
 CLAUDE.md

---
 CLAUDE.md | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/CLAUDE.md b/CLAUDE.md
index c76c212..8e871a4 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -10,6 +10,9 @@ You are a staff-level infrastructure and application engineer/architect. Provide
 - All PR comments must be formatted in markdown
 - Use todo lists to track multi-step tasks
 - **Never add "Generated with Claude Code" or similar attribution lines to PRs or commits**
+- **Never add Co-Authored-By lines to commits**
+- **Always create PRs instead of committing directly to main or protected branches**
+- **Always assign PRs to Noah White**
 
 ## Testing Requirements
 

From 7980063a247c90ab7f782b6b79cc04a95c0cf134 Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sat, 24 Jan 2026 02:59:21 +0000
Subject: [PATCH 07/12] Document feature/** branch naming convention for
 environment access

---
 CLAUDE.md | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index 8e871a4..6d6e62c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -191,10 +191,26 @@ docker restart ghost-compose-caddy-1
 cd /var/mnt/storage/ghost-compose
 ```
 
+## Branch Naming Convention
+
+**All feature branches must follow the `feature/**` pattern** (e.g., `feature/GHO-XX-description`).
+
+This naming convention is required because:
+- The `dev` GitHub environment restricts which branches can access environment-scoped secrets
+- Only `develop` and `feature/**` branches are allowed to access dev environment secrets
+- PR workflows use `environment: dev` to access secrets for `tofu plan` checks
+
+Examples of valid branch names:
+- `feature/GHO-42-add-token-rotation-runbook`
+- `feature/add-new-module`
+- `feature/fix-firewall-rules`
+
+Branches not matching this pattern will fail PR checks due to environment protection rules.
+
 ## Common Tasks
 
 ### Creating a new feature
-1. Create branch from develop: `git checkout -b feature/GHO-XX`
+1. Create branch from develop: `git checkout -b feature/GHO-XX-description`
 2. Make changes
 3. Push and create PR to develop
 4. PR checks run automatically (fmt, plan)

From fed1094d7b534a60faa18a385df64920f49d5413 Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sun, 25 Jan 2026 17:07:08 +0000
Subject: [PATCH 08/12] Use dev-ci shadow environment for PR workflows

- Update pr-tofu-plan-develop.yml to use environment: dev-ci
- Update CLAUDE.md to document dev vs dev-ci environments
- Update token rotation runbook to reflect two-environment setup
---
 .github/workflows/pr-tofu-plan-develop.yml |  2 +-
 CLAUDE.md                                  | 11 +++++------
 docs/token-rotation-runbook.md             | 16 +++++++++-------
 3 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/.github/workflows/pr-tofu-plan-develop.yml b/.github/workflows/pr-tofu-plan-develop.yml
index 43b5eef..2a2d6db 100644
--- a/.github/workflows/pr-tofu-plan-develop.yml
+++ b/.github/workflows/pr-tofu-plan-develop.yml
@@ -14,7 +14,7 @@ permissions:
 jobs:
   tofu-plan:
     runs-on: ubuntu-latest
-    environment: dev
+    environment: dev-ci
 
     steps:
       - name: Install Bitwarden Secrets Manager CLI (bws)
diff --git a/CLAUDE.md b/CLAUDE.md
index 6d6e62c..781dba2 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -195,18 +195,17 @@ cd /var/mnt/storage/ghost-compose
 
 **All feature branches must follow the `feature/**` pattern** (e.g., `feature/GHO-XX-description`).
 
-This naming convention is required because:
-- The `dev` GitHub environment restricts which branches can access environment-scoped secrets
-- Only `develop` and `feature/**` branches are allowed to access dev environment secrets
-- PR workflows use `environment: dev` to access secrets for `tofu plan` checks
+This naming convention is recommended for consistency and traceability.
+
+### GitHub Environments
+- **`dev`**: Protected environment for actual deployments. Only `develop` branch can deploy. Used by `deploy-dev.yml`.
+- **`dev-ci`**: Shadow environment for PR validation. No branch restrictions. Used by `pr-tofu-plan-develop.yml` for `tofu plan` checks. Has required reviewers for security (public repo).
 
 Examples of valid branch names:
 - `feature/GHO-42-add-token-rotation-runbook`
 - `feature/add-new-module`
 - `feature/fix-firewall-rules`
 
-Branches not matching this pattern will fail PR checks due to environment protection rules.
-
 ## Common Tasks
 
 ### Creating a new feature
diff --git a/docs/token-rotation-runbook.md b/docs/token-rotation-runbook.md
index d6d14a7..d5fa6e5 100644
--- a/docs/token-rotation-runbook.md
+++ b/docs/token-rotation-runbook.md
@@ -39,9 +39,10 @@ GitHub secrets are scoped at two levels:
 | Scope | Usage | Naming Convention |
 |-------|-------|-------------------|
 | **Repository-level** | PR workflows (cannot access environment secrets) | `SECRET_NAME_DEV` suffix |
-| **Environment-scoped** | Deploy workflows (`environment: dev`) | `SECRET_NAME` (no suffix) |
+| **Environment-scoped (dev)** | Deploy workflows (`environment: dev`) | `SECRET_NAME` (no suffix) |
+| **Environment-scoped (dev-ci)** | PR workflows (`environment: dev-ci`) | `SECRET_NAME` (no suffix) |
 
-**Important:** When a secret is environment-scoped, you must update it in the GitHub environment settings (Settings → Environments → dev), not in the repository secrets.
+**Important:** When a secret is environment-scoped, you must update it in the GitHub environment settings (Settings → Environments → dev or dev-ci), not in the repository secrets. Secrets shared between `dev` and `dev-ci` must be updated in both environments.
 
 ---
 
@@ -149,9 +150,10 @@ GitHub secrets are scoped at two levels:
 
 **Purpose:** Authenticate to Bitwarden Secrets Manager to retrieve runtime secrets in CI/CD.
 
-**Scope:** Environment-scoped (`environment: dev`)
+**Scope:** Environment-scoped (`dev` and `dev-ci` environments)
 
-Both PR and deploy workflows use `environment: dev`, so a single environment-scoped secret is sufficient.
+- `dev` environment: Used by deploy workflows
+- `dev-ci` environment: Used by PR workflows (shadow environment for validation)
 
 **Expiration:** Machine account tokens do not expire, but rotation is recommended every 6-12 months.
 
@@ -166,10 +168,10 @@ Both PR and deploy workflows use `environment: dev`, so a single environment-sco
    - Name: Include date (e.g., `ci-2025-01`)
    - Copy the token immediately (shown only once)
 
-2. **Update GitHub Secret:**
+2. **Update GitHub Secrets (both environments):**
    - Go to `github.com/noahwhite/ghost-stack` → Settings → Environments → dev
-   - Find `BWS_ACCESS_TOKEN`
-   - Click pencil icon → Update → paste new token → Update secret
+   - Find `BWS_ACCESS_TOKEN` → Update → paste new token
+   - Repeat for Environments → dev-ci
 
 3. **Revoke old token:**
    - In Bitwarden, delete the old access token from the machine account

From 6d04659ee74dd2353926d580f28537c9c5523594 Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sun, 25 Jan 2026 18:30:34 +0000
Subject: [PATCH 09/12] Fix Grafana Cloud token rotation steps with correct
 procedures

- Update access token rotation: use Cloud access policies, correct naming
- Update service account token rotation: correct service account name
- Set expiration to 30 days for both tokens
- Add Bitwarden secret names and notes update instructions
---
 docs/token-rotation-runbook.md | 54 ++++++++++++++++++++++++----------
 1 file changed, 38 insertions(+), 16 deletions(-)

diff --git a/docs/token-rotation-runbook.md b/docs/token-rotation-runbook.md
index d5fa6e5..65adc6a 100644
--- a/docs/token-rotation-runbook.md
+++ b/docs/token-rotation-runbook.md
@@ -67,8 +67,8 @@ GitHub secrets are scoped at two levels:
 | PagerDuty Client ID | PagerDuty | `7d51661b-...` | N/A | N/A | Never |
 | PagerDuty Client Secret | PagerDuty | `b15575c0-...` | N/A | N/A | Never |
 | PagerDuty User Token | PagerDuty | `02805292-...` | N/A | N/A | Never |
-| Grafana Cloud Token | Grafana | `bfc8dd06-...` | N/A | N/A | Never |
-| Grafana Cloud SA Token | Grafana | `3ebc4398-...` | N/A | N/A | Never |
+| Grafana Cloud Token | Grafana | `bfc8dd06-...` | N/A | N/A | 30 days |
+| Grafana Cloud SA Token | Grafana | `3ebc4398-...` | N/A | N/A | 30 days |
 | Linear API Token | Linear | N/A | N/A (local) | N/A | Never |
 | Admin IP | N/A | N/A | `ADMIN_IP` | Environment (dev) | N/A |
 | Cloudflare Zone ID | N/A | N/A | `CLOUDFLARE_ZONE_ID` | Environment (dev) | N/A |
@@ -446,48 +446,70 @@ GitHub secrets are scoped at two levels:
 
 **Purpose:** Configure Grafana Cloud observability via OpenTofu.
 
+**Bitwarden Secret:** `grafana_cloud_access_token`
+
 **Bitwarden Secret ID:** `bfc8dd06-bd97-499a-98f8-b3a101570606`
 
-**Expiration:** Never (but rotation recommended)
+**Expiration:** 30 days
 
 #### Rotation Steps
 
 1. **Generate new token:**
    - Log into Grafana Cloud
-   - Go to My Account → Access Policies
-   - Create new access policy or token with required permissions
-   - Copy the token
+   - Go to Administration → Users and access → Cloud access policies
+   - Find the access policy named `ghost-stack-dev-terraform-token`
+   - Click "Add token"
+   - Name: `soc-dev-grafana-cloud-access-tok-YYYY-MM-DD` (use expiration date)
+   - Set expiry to 30 days
+   - Click "Create"
+   - Copy the token immediately
 
 2. **Update Bitwarden:**
-   - Update secret `bfc8dd06-bd97-499a-98f8-b3a101570606` with new token
+   - Update the secret `grafana_cloud_access_token` with the new token value
+   - Update the notes field with the new expiration date
 
 3. **Revoke old token:**
-   - Delete the old access policy/token in Grafana Cloud
+   - In Grafana Cloud, delete the old token from the access policy
+
+4. **Verify:**
+   - Run `./opentofu/scripts/tofu.sh dev plan`
+   - Confirm Grafana Cloud provider initializes
 
 ---
 
 ### Grafana Cloud Terraform Service Account Token
 
-**Purpose:** Service account for Grafana Cloud Terraform provider (doc project).
+**Purpose:** Service account for Grafana Cloud Terraform provider (soc-dev project).
+
+**Bitwarden Secret:** `grafana_cloud_soc_dev_terraform_sa`
 
 **Bitwarden Secret ID:** `3ebc4398-f4fa-448c-b2c1-b3a6006c063d`
 
-**Expiration:** Configurable
+**Expiration:** 30 days
 
 #### Rotation Steps
 
 1. **Generate new token:**
    - Log into Grafana Cloud
-   - Go to Administration → Service accounts
-   - Find the relevant service account
-   - Create new token
-   - Copy the token
+   - Go to Administration → Users and access → Service Accounts
+   - Find the service account named `sa-1-extsvc-grafana-terraform-app`
+   - Click "Add service account token"
+   - Keep the auto-generated name
+   - Set expiration to 30 days
+   - Click "Generate token"
+   - Copy the token immediately
 
 2. **Update Bitwarden:**
-   - Update secret `3ebc4398-f4fa-448c-b2c1-b3a6006c063d` with new token
+   - Update the secret `grafana_cloud_soc_dev_terraform_sa` with the new token value
+   - Update the comment with the new token name (auto-generated)
+   - Update the notes field with the new expiration date
 
 3. **Delete old token:**
-   - Remove the old token from the service account
+   - In Grafana Cloud, remove the old token from the service account
+
+4. **Verify:**
+   - Run `./opentofu/scripts/tofu.sh dev plan`
+   - Confirm Grafana resources are accessible
 
 ---
 

From 99d058435f693829e8f0790ae482c4428f894fe6 Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sun, 25 Jan 2026 18:31:22 +0000
Subject: [PATCH 10/12] Fix date format to DD-MM-YYYY in Grafana token naming

---
 docs/token-rotation-runbook.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/token-rotation-runbook.md b/docs/token-rotation-runbook.md
index 65adc6a..01fe39c 100644
--- a/docs/token-rotation-runbook.md
+++ b/docs/token-rotation-runbook.md
@@ -459,7 +459,7 @@ GitHub secrets are scoped at two levels:
    - Go to Administration → Users and access → Cloud access policies
    - Find the access policy named `ghost-stack-dev-terraform-token`
    - Click "Add token"
-   - Name: `soc-dev-grafana-cloud-access-tok-YYYY-MM-DD` (use expiration date)
+   - Name: `soc-dev-grafana-cloud-access-tok-DD-MM-YYYY` (use expiration date)
    - Set expiry to 30 days
    - Click "Create"
    - Copy the token immediately

From b459cb6492b78ae46bf67ce767fa33de922f709a Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sun, 25 Jan 2026 18:59:15 +0000
Subject: [PATCH 11/12] Add troubleshooting steps for Grafana token rotation

- Add curl verification commands before saving tokens
- Document common issue: tokens ending with == may be truncated
- Add guidance on token format and length
---
 docs/token-rotation-runbook.md | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/docs/token-rotation-runbook.md b/docs/token-rotation-runbook.md
index 01fe39c..31141ed 100644
--- a/docs/token-rotation-runbook.md
+++ b/docs/token-rotation-runbook.md
@@ -471,10 +471,23 @@ GitHub secrets are scoped at two levels:
 3. **Revoke old token:**
    - In Grafana Cloud, delete the old token from the access policy
 
-4. **Verify:**
+4. **Verify the token before saving:**
+   ```bash
+   curl -H "Authorization: Bearer YOUR_TOKEN" https://grafana.com/api/instances
+   ```
+   - Should return a JSON response with your stacks
+   - If you get 401 Unauthorized, the token is invalid or truncated
+
+5. **Verify after updating Bitwarden:**
    - Run `./opentofu/scripts/tofu.sh dev plan`
    - Confirm Grafana Cloud provider initializes
 
+#### Troubleshooting
+
+- **Token appears truncated:** Grafana tokens are base64-encoded and often end with `=` or `==`. Ensure the entire token was copied including any trailing characters.
+- **401 Unauthorized after rotation:** Verify the token works with the curl command above before saving to Bitwarden. If curl fails, regenerate the token.
+- **Token format:** Valid tokens are typically 50+ characters. If significantly shorter, it was likely truncated during copy.
+
 ---
 
 ### Grafana Cloud Terraform Service Account Token
@@ -507,10 +520,24 @@ GitHub secrets are scoped at two levels:
 3. **Delete old token:**
    - In Grafana Cloud, remove the old token from the service account
 
-4. **Verify:**
+4. **Verify the token before saving:**
+   ```bash
+   curl -H "Authorization: Bearer YOUR_SA_TOKEN" \
+     https://separationofconcerns0dev.grafana.net/api/folders
+   ```
+   - Should return a JSON response with folders
+   - If you get 401 Unauthorized, the token is invalid or truncated
+
+5. **Verify after updating Bitwarden:**
    - Run `./opentofu/scripts/tofu.sh dev plan`
    - Confirm Grafana resources are accessible
 
+#### Troubleshooting
+
+- **Token appears truncated:** Service account tokens are base64-encoded and often end with `=` or `==`. Ensure the entire token was copied.
+- **401 Unauthorized after rotation:** Verify the token works with the curl command above before saving to Bitwarden.
+- **Wrong service account:** Ensure you're creating the token under `sa-1-extsvc-grafana-terraform-app`, not a different service account.
+
 ---
 
 ## Linear API Token

From 8e9217d2a9ff499b741e52c788681f61b4304a4e Mon Sep 17 00:00:00 2001
From: Noah White <noah@noahwhite.net>
Date: Sun, 25 Jan 2026 19:45:46 +0000
Subject: [PATCH 12/12] Enhance Grafana token troubleshooting with more
 specific guidance

- Add token format patterns (glc_* for cloud, glsa_* for service account)
- Recommend using copy button in Grafana UI to avoid truncation
- Clarify that verification should always happen before saving
---
 docs/token-rotation-runbook.md | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/docs/token-rotation-runbook.md b/docs/token-rotation-runbook.md
index 31141ed..41655b2 100644
--- a/docs/token-rotation-runbook.md
+++ b/docs/token-rotation-runbook.md
@@ -484,9 +484,10 @@ GitHub secrets are scoped at two levels:
 
 #### Troubleshooting
 
-- **Token appears truncated:** Grafana tokens are base64-encoded and often end with `=` or `==`. Ensure the entire token was copied including any trailing characters.
-- **401 Unauthorized after rotation:** Verify the token works with the curl command above before saving to Bitwarden. If curl fails, regenerate the token.
-- **Token format:** Valid tokens are typically 50+ characters. If significantly shorter, it was likely truncated during copy.
+- **Token appears truncated:** Grafana cloud access tokens are base64-encoded and often end with `=` or `==`. Ensure the entire token was copied including any trailing characters.
+- **Token format:** Valid cloud access tokens are typically 50+ characters and follow the pattern `glc_xxxx...xxxx==`. If significantly shorter or missing the `==` suffix, it was likely truncated during copy.
+- **401 Unauthorized after rotation:** Always verify the token works with the curl command above before saving to Bitwarden. If curl fails, regenerate the token.
+- **Copy issues:** When copying tokens, use the copy button in the Grafana UI rather than manual selection to avoid truncation.
 
 ---
 
@@ -534,9 +535,11 @@ GitHub secrets are scoped at two levels:
 
 #### Troubleshooting
 
-- **Token appears truncated:** Service account tokens are base64-encoded and often end with `=` or `==`. Ensure the entire token was copied.
-- **401 Unauthorized after rotation:** Verify the token works with the curl command above before saving to Bitwarden.
+- **Token appears truncated:** Service account tokens are base64-encoded and often end with `=` or `==`. Ensure the entire token was copied including any trailing characters.
+- **Token format:** Valid service account tokens are typically 50+ characters and follow the pattern `glsa_xxxx...xxxx==`. If significantly shorter or missing the `==` suffix, it was likely truncated during copy.
+- **401 Unauthorized after rotation:** Always verify the token works with the curl command above before saving to Bitwarden. If curl fails, regenerate the token.
 - **Wrong service account:** Ensure you're creating the token under `sa-1-extsvc-grafana-terraform-app`, not a different service account.
+- **Copy issues:** When copying tokens, use the copy button in the Grafana UI rather than manual selection to avoid truncation.
 
 ---