|
325 | 325 | - file_set_match + provenance: file coverage + citation verification (text must mention file path and repo) |
326 | 326 | - pkg/dataobj/explorer/service.go confirmed via direct read_file: 4 endpoints, no middleware wrapping |
327 | 327 | --- |
| 328 | +[2026-02-20 20:57:58 UTC] Iteration 4 no story markers found |
| 329 | +[2026-02-20 20:57:58 UTC] Iteration 4 complete |
| 330 | +[2026-02-20 20:58:00 UTC] Iteration 5 started |
| 331 | + |
| 332 | +## 2026-02-20 - US-014: Starter tasks — Category D: Incident debugging (1 task) |
| 333 | +- Authored 1 complete task in benchmarks/ccb_mcp_incident/: |
| 334 | + |
| 335 | +**CCX-incident-031** (D31 — error to code path, multi-org-go fixture): |
| 336 | + - Scenario: on-call SRE traces Kubernetes watch-stream error "etcdserver: mvcc: required revision has been compacted" to authoritative etcd source |
| 337 | + - Oracle: {etcd-io/etcd, server/storage/mvcc/kvstore.go, ErrCompacted} + {etcd-io/etcd, server/storage/mvcc/kvstore_txn.go, rangeKeys} |
| 338 | + - Decoy design: kubernetes/kubernetes has vendored copies at vendor/go.etcd.io/etcd/server/v3/storage/mvcc/kvstore.go + apiserver error-mapping at staging/src/k8s.io/apiserver/pkg/storage/etcd3/errors.go — instruction explicitly warns about these to test precision |
| 339 | + - eval.sh: file_set_match (oracle files must match) + keyword_presence ("ErrCompacted", "rangeKeys") |
| 340 | + - Validity gate: VALID (gold=1.0, empty=0.0) |
| 341 | + - Registered in configs/selected_mcp_unique_tasks.json (6 tasks total now) |
| 342 | + |
| 343 | +- Files changed: benchmarks/ccb_mcp_incident/ccx-incident-031/ (8 new files), configs/selected_mcp_unique_tasks.json, prd.json |
| 344 | +- **Learnings for future iterations:** |
| 345 | + - Decoy design principle: use vendored copies in local repo as decoys — they look identical to the real source but are not authoritative |
| 346 | + - "negative knowledge" in instruction.md (explicitly warning about decoys) creates clearer test of whether agent can distinguish source vs. copy |
| 347 | + - kubernetes/kubernetes is large (~800MB) so --depth 1 is essential for Dockerfile clone |
| 348 | + - multi-org-go fixture: kubernetes=local, etcd/grafana=MCP-only — natural for incident debugging where error originates in etcd |
| 349 | + - Error "mvcc: required revision has been compacted" verified: defined in kvstore.go line 37, returned in kvstore_txn.go line 81 (rangeKeys function) |
| 350 | + - oracle_checks.py file_set_match matches on (repo, path) tuples — repo must be "etcd-io/etcd" not "github.com/etcd-io/etcd" |
| 351 | +--- |
0 commit comments