Add Secret Injection Middleware proposal

[jhrozek]

Problem

Many backend services use static authentication (API keys, tokens) instead of OAuth/OIDC. Our token exchange middleware works great for OAuth-compatible backends, but we can't handle scenarios where users need their own individual credentials.

Right now, pkg/secrets resolves secrets once at workload startup and injects them as environment variables. This means secrets are shared across all users - there's no way to isolate credentials per user or track which user accessed what.

Use Cases

Per-User Algolia Admin Keys

When multiple developers manage Algolia search indices through an MCP server, all modifications currently appear to come from one shared account. There's no way to audit who made which changes or enforce individual permissions.

With secret injection, each user's admin key is stored in Vault under their identity. When Alice modifies an index, the middleware authenticates to Vault as Alice and retrieves her specific key. Now we get proper attribution and a complete audit trail.

Multi-Tenant SaaS

Running a single MCP server for multiple customers breaks down when you need tenant-specific credentials. One shared API key for Jira or Salesforce means all tenant data gets accessed through a single account - no isolation.

The solution is storing each tenant's API key in Vault (like secret/tenants/acme-corp/jira-api-key). The user's JWT contains their tenant claim, and the middleware fetches the right credential for that tenant.

Centralized Secret Management

Static secrets in environment variables don't meet enterprise security requirements. Security teams want centralized policies, rotation, and audit logs.

This middleware lets us store API keys only in Vault with proper access controls. The proxy fetches them dynamically, and every access is logged with the user's identity.

How It Works

The middleware sits in the proxy chain after token exchange:

1. Uses the JWT token to authenticate to Vault
2. Fetches the user-specific secret from the configured path
3. Injects it into request headers before forwarding to the backend

In Phase 1, we support static paths. Phase 2 adds Go templating so paths can include user claims like secret/users/{{.email}}/api-key.

Design Decisions

We're using a generic SecretFetcher interface with Vault as the first implementation. This keeps the door open for AWS Secrets Manager or Azure Key Vault later.

The Vault integration uses JWT authentication so each user's identity flows through to secret access. We're implementing the Vault client with direct HTTP calls instead of importing their SDK to avoid BSL 1.1 licensing issues.

Two-phase delivery: static paths first to prove the pattern, then templating for per-user isolation. This gets us something useful quickly while reducing risk.

This is complementary to pkg/secrets, not a replacement. Use pkg/secrets for shared startup credentials like database URLs. Use secret injection for per-user request-time credentials like personal API keys.

What's In The Proposal

• Detailed Vault setup with security controls (bound_issuer, bound_audiences, proper TTLs)
• CLI flag design and Kubernetes CRD examples
• Comparison table showing how this differs from pkg/secrets
• Testing strategy for unit, integration, and operator tests
• Sequence diagrams showing the request flow

Refs: THV-2063 (token exchange middleware)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.25%. Comparing base (5805898) to head (6b450c5).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2357      +/-   ##
==========================================
- Coverage   54.27%   54.25%   -0.02%     
==========================================
  Files         242      242              
  Lines       23446    23446              
==========================================
- Hits        12725    12721       -4     
- Misses       9506     9516      +10     
+ Partials     1215     1209       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.