Description
Add an enable-secret-masking feature flag (alpha) to mask secret values in step stdout/stderr. When enabled, secret values from environment variables (secretKeyRef, envFrom.secretRef) and secret volumes are replaced with *** in logs.
This prevents accidental secret leakage when viewing logs via kubectl logs or similar tools.
Prior Work
@chmouel implemented this in #9359. The PR was fully functional with all CI checks passing but was closed before merging. The implementation should be carried forward, addressing the review feedback.
How it works (from #9359)
- Controller (
pkg/pod/): During pod creation, collects secret values referenced from step secretKeyRef, envFrom.secretRef, and secret volumes. De-duplicates and skips short values (<3 chars). Prepares mask payload as base64(gzip(base64-per-secret lines)) and passes it to an init container via TEKTON_SECRET_MASK_DATA env var.
- Init container (
secret-mask-init subcommand): Decodes/decompresses the env payload and writes a secret-mask file to an emptyDir volume.
- Entrypoint (
cmd/entrypoint/): Loads the secret-mask file and wraps stdout/stderr with a stream-safe masking writer that handles secrets split across write boundaries using a carry buffer.
Key design decisions
- Stream-safe: masking writer buffers up to
largest-secret-len - 1 bytes to handle secrets split across write boundaries
- Secrets <3 chars are skipped (too many false positives)
- Emits a warning when the largest secret exceeds 64KB (as buffering may delay log visibility)
- Not supported on Windows
Caveats
This is not 100% secure. Secret values are still present in the pod spec (accessible via kubectl get pod -o yaml to anyone with pods/get permission). However, it prevents secrets from appearing in:
kubectl logs output
kubectl describe pod output
- Process listings (
ps aux)
Review feedback to address
From the reviews on #9359:
- Shared constants:
SecretMaskDataEnvVar is defined independently in both cmd/entrypoint/subcommands/ and pkg/pod/ — should be consolidated to avoid drift
- Error handling:
secret-mask-init subcommand silently succeeds when called with 0 or 3+ args — should return an error
- Info leak in warning: The delay warning prints the exact byte length of the largest secret to stderr — consider omitting or generalizing
io.Writer contract: Masking writer returns (len(p), err) when underlying write fails — should return (0, err) or the actual bytes written per io.Writer contract
- Security discussion: Secret values end up as plaintext (base64+gzip, not encrypted) in the pod spec env var. This is acknowledged in the PR description as a known limitation — users with
pods/get can already read secrets in the namespace
References
Description
Add an
enable-secret-maskingfeature flag (alpha) to mask secret values in step stdout/stderr. When enabled, secret values from environment variables (secretKeyRef,envFrom.secretRef) and secret volumes are replaced with***in logs.This prevents accidental secret leakage when viewing logs via
kubectl logsor similar tools.Prior Work
@chmouel implemented this in #9359. The PR was fully functional with all CI checks passing but was closed before merging. The implementation should be carried forward, addressing the review feedback.
How it works (from #9359)
pkg/pod/): During pod creation, collects secret values referenced from stepsecretKeyRef,envFrom.secretRef, and secret volumes. De-duplicates and skips short values (<3 chars). Prepares mask payload asbase64(gzip(base64-per-secret lines))and passes it to an init container viaTEKTON_SECRET_MASK_DATAenv var.secret-mask-initsubcommand): Decodes/decompresses the env payload and writes a secret-mask file to an emptyDir volume.cmd/entrypoint/): Loads the secret-mask file and wraps stdout/stderr with a stream-safe masking writer that handles secrets split across write boundaries using a carry buffer.Key design decisions
largest-secret-len - 1bytes to handle secrets split across write boundariesCaveats
This is not 100% secure. Secret values are still present in the pod spec (accessible via
kubectl get pod -o yamlto anyone withpods/getpermission). However, it prevents secrets from appearing in:kubectl logsoutputkubectl describe podoutputps aux)Review feedback to address
From the reviews on #9359:
SecretMaskDataEnvVaris defined independently in bothcmd/entrypoint/subcommands/andpkg/pod/— should be consolidated to avoid driftsecret-mask-initsubcommand silently succeeds when called with 0 or 3+ args — should return an errorio.Writercontract: Masking writer returns(len(p), err)when underlying write fails — should return(0, err)or the actual bytes written perio.Writercontractpods/getcan already read secrets in the namespaceReferences