Skip to content

Add secret masking to step logs #9714

@vdemeester

Description

@vdemeester

Description

Add an enable-secret-masking feature flag (alpha) to mask secret values in step stdout/stderr. When enabled, secret values from environment variables (secretKeyRef, envFrom.secretRef) and secret volumes are replaced with *** in logs.

This prevents accidental secret leakage when viewing logs via kubectl logs or similar tools.

Prior Work

@chmouel implemented this in #9359. The PR was fully functional with all CI checks passing but was closed before merging. The implementation should be carried forward, addressing the review feedback.

How it works (from #9359)

  1. Controller (pkg/pod/): During pod creation, collects secret values referenced from step secretKeyRef, envFrom.secretRef, and secret volumes. De-duplicates and skips short values (<3 chars). Prepares mask payload as base64(gzip(base64-per-secret lines)) and passes it to an init container via TEKTON_SECRET_MASK_DATA env var.
  2. Init container (secret-mask-init subcommand): Decodes/decompresses the env payload and writes a secret-mask file to an emptyDir volume.
  3. Entrypoint (cmd/entrypoint/): Loads the secret-mask file and wraps stdout/stderr with a stream-safe masking writer that handles secrets split across write boundaries using a carry buffer.

Key design decisions

  • Stream-safe: masking writer buffers up to largest-secret-len - 1 bytes to handle secrets split across write boundaries
  • Secrets <3 chars are skipped (too many false positives)
  • Emits a warning when the largest secret exceeds 64KB (as buffering may delay log visibility)
  • Not supported on Windows

Caveats

This is not 100% secure. Secret values are still present in the pod spec (accessible via kubectl get pod -o yaml to anyone with pods/get permission). However, it prevents secrets from appearing in:

  • kubectl logs output
  • kubectl describe pod output
  • Process listings (ps aux)

Review feedback to address

From the reviews on #9359:

  1. Shared constants: SecretMaskDataEnvVar is defined independently in both cmd/entrypoint/subcommands/ and pkg/pod/ — should be consolidated to avoid drift
  2. Error handling: secret-mask-init subcommand silently succeeds when called with 0 or 3+ args — should return an error
  3. Info leak in warning: The delay warning prints the exact byte length of the largest secret to stderr — consider omitting or generalizing
  4. io.Writer contract: Masking writer returns (len(p), err) when underlying write fails — should return (0, err) or the actual bytes written per io.Writer contract
  5. Security discussion: Secret values end up as plaintext (base64+gzip, not encrypted) in the pod spec env var. This is acknowledged in the PR description as a known limitation — users with pods/get can already read secrets in the namespace

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.

    Projects

    Status

    In Progress

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions