test(qa): contention simulation, secret-detection corpus, CI gates#33
Closed
test(qa): contention simulation, secret-detection corpus, CI gates#33
Conversation
Add PostureResult, _find_proof_packs, _run_assay_posture, _post_pr_comment, and emit_posture_comment to assay_bridge.py. Hook into orchestrator at PR_OPEN transition to automatically run `assay posture` and post results as PR comments via `gh pr comment`. Contract: AgentMesh calls Assay, does not reinterpret Assay. Assay owns posture semantics. AgentMesh owns when/where to attach. Requires: assay-toolkit proof posture CLI (`assay posture <pack_dir>`). Expected JSON keys: disposition, claims, residual_risk, proof_debt. Expected exit codes: 0 (verified/supported), 1 (incomplete/blocked), 3 (bad input). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add multi-agent contention simulation tests (same-file claim race, port double-claim, steal-against-fresh-holder, heavy concurrent weave append) and secret-detection corpus tests (6 fixture specimens covering all 7 default private patterns plus clean control). Wire both into PR, nightly, and release CI workflows. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add content_scan_exempt_globs to classifier policy and set tests/fixtures/secrets/** as exempt. Prevents the PR public-private guard from self-blocking on the intentional secret specimens added in the previous commit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Locks the exemption behavior: a file with secret content under an exempt glob is classified PUBLIC, not PRIVATE. Prevents the guard fix from regressing silently. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
7 tasks
There was a problem hiding this comment.
Pull request overview
Adds new QA coverage for multi-agent contention and secret-detection, updates the public/private classifier policy to allow exempting synthetic fixtures from content scanning, and wires these QA slices into CI workflows.
Changes:
- Added contention simulation tests (claim races, steal guard, heavy concurrent weave append).
- Added a secret-detection fixture corpus + regression tests, plus a classifier policy knob to exempt specific paths from content scanning.
- Added/updated CI workflows for PR QA, nightly simulations, and release checks.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_secret_detection.py |
New parameterized tests that validate content-based PRIVATE classification using fixture specimens. |
tests/test_public_private.py |
Adds regression test ensuring content_scan_exempt_globs bypasses content scanning. |
tests/test_contention.py |
New concurrency tests for claim races, steal behavior, and high-contention weave appends. |
tests/fixtures/secrets/private_key.pem |
Synthetic “private key” specimen for content scanner tests. |
tests/fixtures/secrets/pricing_doc.md |
Business-sensitive “pricing” specimen fixture. |
tests/fixtures/secrets/ghp_token.py |
Synthetic GitHub PAT-shaped specimen fixture. |
tests/fixtures/secrets/edge_ghp_in_comment.py |
Fixture ensuring token detection also triggers from comments. |
tests/fixtures/secrets/clean_public.py |
Clean control fixture expected to remain PUBLIC. |
tests/fixtures/secrets/aws_key.py |
Synthetic AWS access key ID specimen fixture. |
src/agentmesh/public_private.py |
Adds content_scan_exempt_globs support to skip content scanning on matching paths. |
src/agentmesh/orchestrator.py |
Adds PR_OPEN transition side effect to emit an Assay “proof posture” PR comment. |
src/agentmesh/assay_bridge.py |
Implements posture evaluation via assay posture and PR commenting via gh. |
.github/workflows/release-agentmesh-check.yml |
New release workflow to build/install wheel and run full tests. |
.github/workflows/pr-agentmesh-qa.yml |
New PR workflow to run contention + secret-detection tests. |
.github/workflows/nightly-agentmesh-simulations.yml |
New nightly workflow to run full suite across Python versions + heavier contention slice. |
.agentmesh/policy.json |
Adds repo policy exemption for tests/fixtures/secrets/** to prevent self-blocking. |
You can also share your feedback on Copilot code review. Take the survey.
Comment on lines
+136
to
+149
| def test_heavy_concurrent_weave_append(tmp_data_dir: Path) -> None: | ||
| """50 concurrent writers via ThreadPoolExecutor(max_workers=16). | ||
|
|
||
| All must get unique monotonic sequence IDs, no gaps, verify_weave() passes. | ||
| """ | ||
| n = 50 | ||
|
|
||
| def _append(i: int) -> int: | ||
| evt = append_weave(capsule_id=f"heavy_{i}", data_dir=tmp_data_dir) | ||
| return evt.sequence_id | ||
|
|
||
| with ThreadPoolExecutor(max_workers=16) as pool: | ||
| seqs = list(pool.map(_append, range(n))) | ||
|
|
Comment on lines
+1
to
+4
| -----BEGIN RSA PRIVATE KEY----- | ||
| MIIBogIBAAJBALRiMLAHudeSA/x3hB2f+2NRkJLA/FAKEFAKEFAKEFAKEFAKE | ||
| FAKEFAKEFAKEFAKEFAKEFAKEFAKEFAKEFAKEFAKEFAKEFAKEFAKEFAKE1234 | ||
| -----END RSA PRIVATE KEY----- |
| @@ -0,0 +1,2 @@ | |||
| # This file contains a leaked AWS access key for testing secret detection. | |||
| AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE" | |||
Comment on lines
+295
to
+308
| def _post_pr_comment(pr_ref: str, body: str, repo_path: Path) -> bool: | ||
| """Post a comment to a PR via ``gh pr comment``. Returns True on success.""" | ||
| if shutil.which("gh") is None: | ||
| return False | ||
| try: | ||
| subprocess.run( | ||
| ["gh", "pr", "comment", pr_ref, "--body", body], | ||
| capture_output=True, | ||
| text=True, | ||
| timeout=30, | ||
| cwd=str(repo_path), | ||
| ) | ||
| return True | ||
| except Exception: |
Comment on lines
+378
to
+384
| "task_id": task_id, | ||
| "action": "posture_comment", | ||
| "pr_ref": pr_ref, | ||
| "bridge_status": result.status, | ||
| "disposition": posture.get("disposition", ""), | ||
| "pack_dir": str(pack_dir) if pack_dir else "", | ||
| "degraded_reason": result.reason, |
Comment on lines
+479
to
+491
| # PR_OPEN side effect: emit proof posture comment if PR is available. | ||
| # AgentMesh calls Assay, does not reinterpret Assay. | ||
| if to_state == TaskState.PR_OPEN: | ||
| pr_ref = task.pr_url or task.branch | ||
| if pr_ref: | ||
| try: | ||
| assay_bridge.emit_posture_comment( | ||
| task_id=task_id, | ||
| pr_ref=pr_ref, | ||
| agent_id=agent_id, | ||
| episode_id=task.episode_id, | ||
| data_dir=data_dir, | ||
| ) |
Comment on lines
+1
to
+2
| # This file contains a leaked GitHub PAT for testing secret detection. | ||
| API_TOKEN = "ghp_R8x2mN4vL6pQ9wK1jT3yF5bA7cE0hU2sG4nM" |
Comment on lines
+122
to
+123
| (repo / "tests" / "fixtures" / "secrets" / "token.py").write_text( | ||
| 'API_TOKEN = "ghp_R8x2mN4vL6pQ9wK1jT3yF5bA7cE0hU2sG4nM"\n' |
Comment on lines
+312
to
+327
| def emit_posture_comment( | ||
| *, | ||
| task_id: str, | ||
| pr_ref: str, | ||
| repo_path: Path | None = None, | ||
| pack_dir: Path | None = None, | ||
| require_falsifiers: bool = False, | ||
| agent_id: str = "", | ||
| episode_id: str = "", | ||
| data_dir: Path | None = None, | ||
| ) -> PostureResult: | ||
| """Run proof posture and post a PR comment. | ||
|
|
||
| AgentMesh calls Assay, does not reinterpret Assay. | ||
| Assay owns posture semantics. AgentMesh owns when/where to attach. | ||
| """ |
Comment on lines
+484
to
+493
| try: | ||
| assay_bridge.emit_posture_comment( | ||
| task_id=task_id, | ||
| pr_ref=pr_ref, | ||
| agent_id=agent_id, | ||
| episode_id=task.episode_id, | ||
| data_dir=data_dir, | ||
| ) | ||
| except Exception: | ||
| pass # Posture is best-effort, never blocks transition |
Owner
Author
|
Superseded by #34 — QA commits split to a clean branch off main. This branch also carries proof-posture bridge changes that belong in a separate PR. |
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
content_scan_exempt_globsto classifier policy so synthetic test fixtures don't self-block the PR public-private guardWhat changed
ba8695ctest_contention.py,test_secret_detection.py, 6 fixture files, 3 workflow YAMLs71ad32acontent_scan_exempt_globsinpublic_private.py+policy.json7872ee5Test plan
🤖 Generated with Claude Code