feat: secrets filtering and sensitive data protection (Issue #12)

[zircote]

Summary

Implements comprehensive secrets detection and filtering to prevent sensitive data (API keys, passwords, tokens, private keys, PII) from being captured in memories.

GitHub Issue: #12

Approach

• Uses detect-secrets (Yelp) library for battle-tested secret detection (27+ patterns)
• Custom PII detection for SSN, credit cards, phone numbers
• Default REDACT strategy (configurable: MASK, BLOCK, WARN)
• Hash-based allowlist for known-safe values
• Full audit logging for SOC2/GDPR compliance

Components

1. SecretsFilteringService - Main orchestrator
2. DetectSecretsAdapter - Wraps detect-secrets library
3. PIIDetector - Custom PII patterns
4. AllowlistManager - Hash-based known-safe values
5. Redactor - Applies filtering strategies
6. AuditLogger - Compliance logging

Implementation Plan

• Phase 1: Foundation (module structure, models, config)
• Phase 2: Detection Layer (detect-secrets, PII)
• Phase 3: Integration (redactor, allowlist, capture pipeline)
• Phase 4: Commands & Audit (CLI, logging)
• Phase 5: Testing & Documentation

Specification

See docs/spec/active/2025-12-25-secrets-filtering/ for full spec:

• REQUIREMENTS.md (14 P0, 6 P1, 5 P2 requirements)
• ARCHITECTURE.md (6 components)
• IMPLEMENTATION_PLAN.md (30 tasks, 5 phases)
• DECISIONS.md (12 ADRs)

Test Plan

• Unit tests for all components (>90% coverage)
• Integration tests for full pipeline
• Performance tests (<10ms latency target)
• Retrospective scanning tests

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR implements comprehensive secrets detection and filtering to prevent sensitive data (API keys, passwords, tokens, PII) from being captured in memories. The implementation follows a detailed specification and includes:

• Multi-layer detection: detect-secrets library (27+ patterns) + custom PII detection (SSN, credit cards, phones)
• Flexible filtering strategies: REDACT, MASK, BLOCK, WARN with per-namespace configuration
• Hash-based allowlist: Known-safe values can bypass filtering
• Full audit logging: SOC2/GDPR compliance with JSON Lines format
• Retrospective scanning: Scan existing memories for secrets

Reviewed changes

Copilot reviewed 46 out of 47 changed files in this pull request and generated 24 comments.

Show a summary per file

File	Description
src/git_notes_memory/security/*.py	Complete security module with 9 components
tests/security/*.py	Comprehensive test suite (>90% coverage target)
src/git_notes_memory/capture.py	Integration with capture pipeline for filtering
src/git_notes_memory/index.py	Added get_all_memories() for retrospective scanning
src/git_notes_memory/exceptions.py	Renamed MemoryError → MemoryPluginError
pyproject.toml, uv.lock	Added detect-secrets>=1.4.0 dependency
docs/spec/active/2025-12-25-secrets-filtering/*	Full specification docs

[Copilot]

Variable audit_logger is not used.

[Copilot]

The name 'get_secrets_filtering_service' is exported by all but is not defined.

[Copilot]

The name 'get_secrets_filtering_service' is exported by all but is not defined.

[Copilot]

The name 'get_redactor' is exported by all but is not defined.

Suggested change

	from __future__ import annotations
	__all__ = [
	# Factory functions (lazy-loaded)
	"get_secrets_filtering_service",
	"get_redactor",
	"get_allowlist_manager",
	"get_audit_logger",
	# Enums
	"SecretType",
	"FilterStrategy",
	"FilterAction",
	# Models
	"SecretDetection",
	"FilterResult",
	"AllowlistEntry",
	"AuditEntry",
	# Exceptions
	"SecretsFilteringError",
	"BlockedContentError",
	"AllowlistError",
	"AuditLogError",
	]
	def __getattr__(name: str) -> object:
	from __future__ import annotations
	from typing import Any
	__all__ = [
	# Factory functions (lazy-loaded)
	"get_secrets_filtering_service",
	"get_redactor",
	"get_allowlist_manager",
	"get_audit_logger",
	# Enums
	"SecretType",
	"FilterStrategy",
	"FilterAction",
	# Models
	"SecretDetection",
	"FilterResult",
	"AllowlistEntry",
	"AuditEntry",
	# Exceptions
	"SecretsFilteringError",
	"BlockedContentError",
	"AllowlistError",
	"AuditLogError",
	]
	def get_redactor(*args: Any, **kwargs: Any) -> Any:
	"""Return the default redactor instance.
	This wrapper exposes ``get_redactor`` as a real module attribute while
	delegating to the lazily-loaded implementation resolved via
	``__getattr__`` to avoid importing heavy dependencies at import time.
	"""
	redactor_factory = __getattr__("get_redactor")
	return redactor_factory(*args, **kwargs)
	def __getattr__(name: str) -> object:

[Copilot]

The name 'get_allowlist_manager' is exported by all but is not defined.

[Copilot]

User input from the /memory:test-secret command is interpolated directly into a Python -c program via test_value = '''${TEST_VALUE}''', so a crafted value containing ''' and extra Python code can break out of the string literal and execute arbitrary code in the plugin process. An attacker could convince a developer to run /memory:test-secret with a malicious argument (e.g., copied from untrusted documentation), leading to arbitrary Python execution with the plugin's local file and environment access. To fix this, avoid embedding raw arguments into the Python source and instead pass them via sys.argv or environment variables and parse them inside Python, or at minimum serialize/escape the value safely (e.g., using JSON) before interpolation so it cannot alter the program structure.

Suggested change

	import sys
	from git_notes_memory.security import get_secrets_filtering_service
	from git_notes_memory.security.models import FilterAction
	# Get the test value - passed as environment or inline
	test_value = '''${TEST_VALUE}'''
	import os
	import sys
	from git_notes_memory.security import get_secrets_filtering_service
	from git_notes_memory.security.models import FilterAction
	# Get the test value from the TEST_VALUE environment variable
	test_value = os.environ.get('TEST_VALUE', '')

[Copilot]

The /memory:secrets-allowlist add subcommand embeds user-supplied --hash, --namespace, and --reason values directly into a Python -c string as hash_val = '${HASH}', namespace = '${NAMESPACE}' or None, and reason = '${REASON}', allowing injected quotes and Python code to break out of the string literal and execute. A malicious value such as a crafted --reason could cause arbitrary Python execution when a developer runs the command, compromising local files or credentials. Refactor this to pass arguments into Python via sys.argv or environment variables and treat them as data (or safely serialize/escape them) rather than concatenating them into executable source code.

Suggested change

	uv run --directory "$PLUGIN_ROOT" python3 -c "
	import sys
	from git_notes_memory.security import get_allowlist_manager, get_audit_logger
	hash_val = '${HASH}'
	namespace = '${NAMESPACE}' or None
	reason = '${REASON}'
	HASH="$HASH" NAMESPACE="$NAMESPACE" REASON="$REASON" uv run --directory "$PLUGIN_ROOT" python3 -c "
	import os
	import sys
	from git_notes_memory.security import get_allowlist_manager, get_audit_logger
	hash_val = os.environ.get('HASH') or ''
	namespace = os.environ.get('NAMESPACE') or None
	reason = os.environ.get('REASON') or ''

[Copilot]

The /memory:secrets-allowlist remove subcommand similarly interpolates the user-controlled --hash value into the Python -c source as hash_val = '${HASH}', so a hash argument containing quotes and code can escape the string literal and run arbitrary Python. This means a crafted command line (for example copied from untrusted documentation) can trigger code execution in the plugin environment. Use safe argument passing (e.g., sys.argv or environment variables) or robust serialization/escaping instead of embedding raw CLI values into Python source.

Suggested change

	import sys
	from git_notes_memory.security import get_allowlist_manager, get_audit_logger
	hash_val = '${HASH}'
	import os
	import sys
	from git_notes_memory.security import get_allowlist_manager, get_audit_logger
	hash_val = os.environ.get('HASH', '')

[Copilot]

In the first /memory:scan-secrets scan step, the --namespace argument is injected into the Python -c command as namespace_filter = '${NAMESPACE}' or None, which allows a value containing quotes and additional Python code to escape the string and execute in the scanning process. This exposes arbitrary code execution when a developer runs the scan command with a crafted namespace argument. Instead, pass the namespace as data (e.g., via sys.argv or environment) and avoid direct interpolation into the Python program text or ensure it is safely serialized/escaped before use.

Suggested change

	from git_notes_memory import get_recall_service
	from git_notes_memory.security import get_secrets_filtering_service
	from git_notes_memory.security.models import FilterAction
	namespace_filter = '${NAMESPACE}' or None
	import os
	from git_notes_memory import get_recall_service
	from git_notes_memory.security import get_secrets_filtering_service
	from git_notes_memory.security.models import FilterAction
	namespace_filter = os.environ.get('NAMESPACE') or None

[Copilot]

In the remediation step of /memory:scan-secrets, user-controlled --namespace and --dry-run flags are embedded directly into the Python -c code (namespace_filter = '${NAMESPACE}' or None and dry_run = '${DRY_RUN}' == 'true'), so crafted values containing quotes and Python syntax can break out of the string literal and execute. This again provides an arbitrary code execution path whenever a developer invokes the command with specially formed arguments. Mitigate this by treating these flags purely as data passed via sys.argv/environment and parsed in Python, or by serializing/escaping them safely before interpolation so they cannot alter the code structure.

Suggested change

	from git_notes_memory import get_recall_service, get_capture_service
	from git_notes_memory.security import get_secrets_filtering_service, get_audit_logger
	from git_notes_memory.index import IndexService
	from git_notes_memory.config import get_project_index_path
	namespace_filter = '${NAMESPACE}' or None
	dry_run = '${DRY_RUN}' == 'true'
	import os
	from git_notes_memory import get_recall_service, get_capture_service
	from git_notes_memory.security import get_secrets_filtering_service, get_audit_logger
	from git_notes_memory.index import IndexService
	from git_notes_memory.config import get_project_index_path
	namespace_filter = os.environ.get('NAMESPACE') or None
	dry_run = os.environ.get('DRY_RUN', 'false').lower() == 'true'