Skip to content

feat: runtime credential rotation for AMQP, Redis, and TLS#59

Open
markusnissl wants to merge 1 commit intomainfrom
feat/credential-rotation
Open

feat: runtime credential rotation for AMQP, Redis, and TLS#59
markusnissl wants to merge 1 commit intomainfrom
feat/credential-rotation

Conversation

@markusnissl
Copy link
Copy Markdown
Contributor

Summary

  • config.SecretWatcher — watches _FILE env var paths via fsnotify, updates Watchable[string] on content change (Kubernetes symlink-swap pattern)
  • AMQP UpdateURL / UpdateTLS — stores new credentials and triggers reconnect; dial() reads under RLock for thread safety
  • Redis SwapClient — creates new client, validates via ping, atomically swaps, closes old client; fires onReconnect for subscriber refresh
  • TLSReloader — watches cert/key/CA files, provides dynamic GetCertificate/GetClientCertificate callbacks; debounced reload
  • WithSecretRotation() on Builder — single toggle that enables file watchers for all configured infrastructure
  • Database — documented why runtime rotation is infeasible (database/sql pool); recommends Kubernetes rolling restart

How It Works

K8s secret update → mounted file changes (symlink swap)
                   → fsnotify detects change
                   → SecretWatcher reads new content
                   → Watchable.OnChange fires
                   → Component.UpdateURL / SwapClient / TLSReloader.reload
                   → New connections use new credentials

Usage

app.New("my-service", version, cfg.BaseConfig).
    WithRabbitMQ(cfg.AMQPURL).
    WithRedis(&redis.Options{Addr: cfg.RedisAddr}).
    WithSecretRotation().  // ← enables all watchers
    Router(func(infra app.Infrastructure) http.Handler { ... }).
    Run()

What Rotates vs What Doesn't

Component Rotation Mechanism
AMQP URL/password Live UpdateURL → reconnect loop
AMQP TLS certs Live UpdateTLS → reconnect loop
Redis password Live SwapClient → new client + ping validation
TLS certs (HTTP, mTLS) Live TLSReloaderGetCertificate callbacks
Database password Restart database/sql pool can't swap credentials

Test plan

  • SecretWatcher: file change detection, atomic file replace, bad file keeps old value, no-op when no _FILE set
  • TLSReloader: initial load, cert rotation detection, bad cert keeps old
  • AMQP: UpdateURL builds/tests pass, dial() reads under lock
  • Redis: SwapClient builds/tests pass, Client() reads under lock
  • App: builds with WithSecretRotation wiring
  • Integration: end-to-end rotation with real AMQP/Redis (requires Docker)

Add infrastructure for detecting and reacting to secret file changes at
runtime, enabling zero-downtime credential rotation in Kubernetes and
Docker environments.

Core:
- Add config.GetSecretPath to expose _FILE env var paths
- Add config.SecretWatcher that watches _FILE paths via fsnotify and
  updates a Watchable[string] when content changes

AMQP:
- Add Connection.UpdateURL that stores new URL and triggers reconnect
- Add Connection.UpdateTLS for certificate rotation
- Protect dial() reads with RLock for concurrent UpdateURL safety

Redis:
- Add Connection.SwapClient that creates new client with updated
  credentials, validates via ping, atomically swaps, closes old client
- Protect c.client reads in Client(), checkHealth(), Close()
- Fire onReconnect after swap so subscribers refresh references

Database:
- Add gormdb.SwappablePool — sits between GORM and *sql.DB, allows
  atomic pool swap without changing the *gorm.DB pointer
- PostgreSQL driver stores SwappablePool for credential rotation
- New queries immediately use new pool; old pool drains via grace period
- No code changes needed in repositories

TLS:
- Add netutil.TLSReloader that watches cert/key/CA files via fsnotify
- Provides dynamic GetCertificate/GetClientCertificate callbacks
- Atomically reloads all three files together with debounce

Builder:
- Add WithSecretRotation() that enables all watchers with single toggle
- Wire SecretWatcher into messaging and redis modules
- AMQP watches RABBITMQ_URL_FILE or RABBITMQ_PASSWORD_FILE
- Redis watches REDIS_PASSWORD_FILE
@markusnissl markusnissl force-pushed the feat/credential-rotation branch from b9500c9 to a961e22 Compare April 6, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant