|
22 | 22 | {"actor":"sjarmak","comment":null,"created_at":"2026-03-09T13:11:58Z","event_type":"created","id":22,"issue_id":"CodeScaleBench-2kz","new_value":"","old_value":""} |
23 | 23 | {"actor":"sjarmak","comment":null,"created_at":"2026-03-09T14:10:34Z","event_type":"status_changed","id":23,"issue_id":"CodeScaleBench-2kz","new_value":"{\"notes\":\"2026-03-09 validation pass:\\\\n- Fixed stale task generators/templates so fresh org + SDLC scaffolded tasks now render and smoke clean without one-off harness patches.\\\\n- Temp scaffold validation: org template path renders, contract-check passes, and baseline/sg_only smoke runs produce reward artifacts as expected; feature/refactor scaffold outputs pass contract-only plus baseline/sg_only no-agent smoke.\\\\n- Curated local smoke subsets all passed via exact-selection flow: baseline (ccx-onboard-search-207, element-web-unread-indicators-diverge-fix-001, clickhouse-mergetree-arch-understand-001), sg_only (same trio), artifact_only (ccx-onboard-search-207, bustub-hyperloglog-impl-001, nodebb-plugin-validate-fix-001).\\\\n- Prepared rerun manifests: configs/claude_historical_failure_rerun_mcp_20260309.json and configs/openhands_historical_failure_rerun_baseline_20260309.json.\\\\n- Infra readiness checked: account_health.py status recommends proceed; check_infra.py now passes in current workspace.\\\\nRemaining: launch rerun manifests only after interactive confirmation, then classify any residual failures and decide permanent sentinel coverage.\",\"status\":\"in_progress\"}","old_value":"{\"id\":\"CodeScaleBench-2kz\",\"title\":\"Verify harness fixes by rerunning historical Claude/OpenHands failures\",\"description\":\"Run a focused verification batch to prove the current task-contract and harness hardening eliminates the earlier random patch churn.\\n\\nScope:\\n- Claude Code regression sentinels:\\n - mcp_ccx-onboard-search-207\\n - mcp_ccx-onboard-search-208\\n - mcp_ccx-onboard-search-210\\n - mcp_bustub-hyperloglog-impl-001\\n - mcp_django-sensitive-file-exclusion-001\\n - mcp_flink-window-late-data-fix-001\\n - mcp_element-web-unread-indicators-diverge-fix-001\\n - clickhouse-mergetree-arch-understand-001 (confirm Daytona/local routing now that storage metadata was corrected)\\n- OpenHands regression sentinel:\\n - ccx-onboard-search-212\\n\\nAcceptance criteria:\\n- Produce a small rerun manifest or manifests for the tasks above.\\n- Execute the reruns once accounts are ready.\\n- Confirm whether each task now completes as a valid run without ad hoc task-specific patches.\\n- Record any remaining failures as either harness bugs, task bugs, or infra issues with exact root cause.\\n- If clean, note which tasks should remain in the smoke/verification matrix as permanent regression sentinels.\\n\",\"status\":\"open\",\"priority\":1,\"issue_type\":\"task\",\"owner\":\"sjarmak@users.noreply.github.com\",\"created_at\":\"2026-03-09T13:11:58Z\",\"created_by\":\"sjarmak\",\"updated_at\":\"2026-03-09T13:11:58Z\"}"} |
24 | 24 | {"actor":"sjarmak","comment":null,"created_at":"2026-03-09T14:29:12Z","event_type":"updated","id":24,"issue_id":"CodeScaleBench-2kz","new_value":"{\"notes\":\"2026-03-09 validation pass:\\\\n- Fixed stale task generators/templates so fresh org + SDLC scaffolded tasks now render and smoke clean without one-off harness patches.\\\\n- Temp scaffold validation: org template path renders, contract-check passes, and baseline/sg_only smoke runs produce reward artifacts as expected; feature/refactor scaffold outputs pass contract-only plus baseline/sg_only no-agent smoke.\\\\n- Curated local smoke subsets all passed via exact-selection flow: baseline (ccx-onboard-search-207, element-web-unread-indicators-diverge-fix-001, clickhouse-mergetree-arch-understand-001), sg_only (same trio), artifact_only (ccx-onboard-search-207, bustub-hyperloglog-impl-001, nodebb-plugin-validate-fix-001).\\\\n- Prepared rerun manifests: configs/claude_historical_failure_rerun_mcp_20260309.json and configs/openhands_historical_failure_rerun_baseline_20260309.json.\\\\n- Infra readiness checked: account_health.py status recommends proceed; check_infra.py now passes in current workspace.\\\\nRemaining: launch rerun manifests only after interactive confirmation, then classify any residual failures and decide permanent sentinel coverage.\\n2026-03-09 launch started after explicit confirmation.\\\\n- Claude MCP rerun batch launched via configs/run_selected_tasks.sh in Daytona mode using accounts account1/account2/account4 (account3 held, account5 reserved for OpenHands). Run dirs are rooted at runs/staging/csb_org_onboarding_sonnet_20260309_142738, runs/staging/csb_sdlc_feature_sonnet_20260309_142738, runs/staging/csb_sdlc_fix_sonnet_20260309_142738, runs/staging/csb_sdlc_secure_sonnet_20260309_142738, runs/staging/csb_sdlc_understand_sonnet_20260309_142738 under config mcp-remote-direct. Initial live tasks confirmed on disk for ccx-onboard-search-207/208/210.\\\\n- OpenHands baseline sentinel launched via configs/openhands_2config.sh in Daytona mode using account5 only. Run dir: runs/staging/openhands_sonnet46_20260309_142733/baseline-local-direct/.../ccx-onboard-search-212__CDJ962t.\\\\n- Remaining Claude tasks will submit as the 3-slot queue drains.\\\\nNext: monitor task completion/invalids, classify any residual failures, and decide which sentinels stay in permanent smoke coverage.\"}","old_value":"{\"id\":\"CodeScaleBench-2kz\",\"title\":\"Verify harness fixes by rerunning historical Claude/OpenHands failures\",\"description\":\"Run a focused verification batch to prove the current task-contract and harness hardening eliminates the earlier random patch churn.\\n\\nScope:\\n- Claude Code regression sentinels:\\n - mcp_ccx-onboard-search-207\\n - mcp_ccx-onboard-search-208\\n - mcp_ccx-onboard-search-210\\n - mcp_bustub-hyperloglog-impl-001\\n - mcp_django-sensitive-file-exclusion-001\\n - mcp_flink-window-late-data-fix-001\\n - mcp_element-web-unread-indicators-diverge-fix-001\\n - clickhouse-mergetree-arch-understand-001 (confirm Daytona/local routing now that storage metadata was corrected)\\n- OpenHands regression sentinel:\\n - ccx-onboard-search-212\\n\\nAcceptance criteria:\\n- Produce a small rerun manifest or manifests for the tasks above.\\n- Execute the reruns once accounts are ready.\\n- Confirm whether each task now completes as a valid run without ad hoc task-specific patches.\\n- Record any remaining failures as either harness bugs, task bugs, or infra issues with exact root cause.\\n- If clean, note which tasks should remain in the smoke/verification matrix as permanent regression sentinels.\\n\",\"notes\":\"2026-03-09 validation pass:\\\\n- Fixed stale task generators/templates so fresh org + SDLC scaffolded tasks now render and smoke clean without one-off harness patches.\\\\n- Temp scaffold validation: org template path renders, contract-check passes, and baseline/sg_only smoke runs produce reward artifacts as expected; feature/refactor scaffold outputs pass contract-only plus baseline/sg_only no-agent smoke.\\\\n- Curated local smoke subsets all passed via exact-selection flow: baseline (ccx-onboard-search-207, element-web-unread-indicators-diverge-fix-001, clickhouse-mergetree-arch-understand-001), sg_only (same trio), artifact_only (ccx-onboard-search-207, bustub-hyperloglog-impl-001, nodebb-plugin-validate-fix-001).\\\\n- Prepared rerun manifests: configs/claude_historical_failure_rerun_mcp_20260309.json and configs/openhands_historical_failure_rerun_baseline_20260309.json.\\\\n- Infra readiness checked: account_health.py status recommends proceed; check_infra.py now passes in current workspace.\\\\nRemaining: launch rerun manifests only after interactive confirmation, then classify any residual failures and decide permanent sentinel coverage.\",\"status\":\"in_progress\",\"priority\":1,\"issue_type\":\"task\",\"owner\":\"sjarmak@users.noreply.github.com\",\"created_at\":\"2026-03-09T13:11:58Z\",\"created_by\":\"sjarmak\",\"updated_at\":\"2026-03-09T14:10:34Z\"}"} |
| 25 | +{"actor":"sjarmak","comment":null,"created_at":"2026-03-09T15:53:14Z","event_type":"created","id":25,"issue_id":"CodeScaleBench-25b","new_value":"","old_value":""} |
| 26 | +{"actor":"sjarmak","comment":null,"created_at":"2026-03-09T15:53:28Z","event_type":"updated","id":26,"issue_id":"CodeScaleBench-25b","new_value":"{\"description\":\"Goal\\nThe canonical 275-task benchmark should expose a consistent hybrid evaluation contract: every task keeps its deterministic verifier, and tasks that support artifact evaluation must map answer.json into the same reporting model rather than creating incomparable ad hoc reward semantics.\\n\\nWhy this exists\\nThe current canonical set is not uniformly dual-mode. A local audit of configs/selected_benchmark_tasks.json found:\\n- 275 canonical tasks total\\n- 233/275 have Dockerfile.artifact_only\\n- 144/275 also have Dockerfile.artifact_baseline\\n- 89/275 include answer_json_verifier_lib.sh\\n- 136/275 explicitly instruct agents to write answer.json\\n- 42/275 canonical tasks currently lack artifact_only support entirely\\n\\nCurrent shape\\n- Org suites are close to dual-mode: all selected org tasks have artifact_only and artifact_baseline variants.\\n- SDLC suites are mixed: deterministic verifier coverage is effectively universal, but answer.json/artifact support is partial and family-specific.\\n- Reward semantics vary by verifier family (oracle checks, checklist, grep/heuristic repo verification, F1 hybrids, test-ratio), and many reports currently collapse that into a single scalar.\\n\\nDesired end state\\n- Canonical benchmark policy explicitly supports a hybrid of deterministic verifier reward and answer.json-derived reward.\\n- Every canonical task declares evaluator family, expected output artifact(s), and reward interpretation.\\n- Every canonical verifier emits a standard validation_result payload in addition to reward.txt.\\n- Reports distinguish continuous reward from pass/solved status and avoid treating unlike scorer families as directly comparable without calibration.\\n- Remaining missing artifact-mode support is either implemented or intentionally excluded with documented rationale.\\n\\nSuggested child workstreams\\n- Audit and classify the 275 canonical tasks by evaluator family and output contract.\\n- Define the common validation_result schema and pass-threshold semantics.\\n- Close artifact-mode coverage gaps for canonical tasks, or mark intentional exceptions.\\n- Update report generation to surface scorer family, pass/fail, and sub-scores.\\n- Document the hybrid evaluation policy in docs/reference and ops docs.\\n\"}","old_value":"{\"id\":\"CodeScaleBench-25b\",\"title\":\"Unify canonical task evaluation contract across deterministic and answer.json modes\",\"acceptance_criteria\":\"1. Canonical metadata and docs state the supported evaluator family, expected agent output artifact(s), and reward semantics for all 275 selected tasks. 2. Every canonical task emits a standard validation payload in addition to /logs/verifier/reward.txt, including scorer family, sub-scores, pass_threshold, passed, and error/failure context. 3. The canonical benchmark defines an explicit hybrid evaluation policy covering deterministic verifier reward and answer.json-derived reward, with family-specific scoring documented and reportable. 4. All 275 canonical tasks are audited for artifact-mode support; gaps are either fixed or tracked as child beads with clear remediation plans. 5. Reporting distinguishes continuous reward from solved/pass status and does not treat different scorer families as directly comparable without calibration.\",\"status\":\"open\",\"priority\":1,\"issue_type\":\"epic\",\"owner\":\"sjarmak@users.noreply.github.com\",\"created_at\":\"2026-03-09T15:53:14Z\",\"created_by\":\"sjarmak\",\"updated_at\":\"2026-03-09T15:53:14Z\"}"} |
| 27 | +{"actor":"sjarmak","comment":null,"created_at":"2026-03-09T16:05:19Z","event_type":"created","id":27,"issue_id":"CodeScaleBench-25b.1","new_value":"","old_value":""} |
| 28 | +{"actor":"sjarmak","comment":null,"created_at":"2026-03-09T16:05:19Z","event_type":"created","id":28,"issue_id":"CodeScaleBench-25b.2","new_value":"","old_value":""} |
| 29 | +{"actor":"sjarmak","comment":null,"created_at":"2026-03-09T16:05:19Z","event_type":"created","id":29,"issue_id":"CodeScaleBench-25b.3","new_value":"","old_value":""} |
| 30 | +{"actor":"sjarmak","comment":null,"created_at":"2026-03-09T16:05:19Z","event_type":"created","id":30,"issue_id":"CodeScaleBench-25b.4","new_value":"","old_value":""} |
| 31 | +{"actor":"sjarmak","comment":null,"created_at":"2026-03-09T16:05:19Z","event_type":"created","id":31,"issue_id":"CodeScaleBench-25b.5","new_value":"","old_value":""} |
| 32 | +{"actor":"sjarmak","comment":null,"created_at":"2026-03-09T16:11:44Z","event_type":"claimed","id":32,"issue_id":"CodeScaleBench-25b.1","new_value":"{\"assignee\":\"sjarmak\",\"status\":\"in_progress\"}","old_value":"{\"id\":\"CodeScaleBench-25b.1\",\"title\":\"Audit canonical task evaluator families and output contracts\",\"description\":\"Goal\\nProduce a canonical audit of the 275 selected tasks so follow-on work is driven by facts instead of assumptions.\\n\\nScope\\n- Classify each task by verifier family (oracle-checks, checklist, repo-state heuristic, test-ratio, F1-hybrid, etc.).\\n- Record expected agent output contract (solution.json, answer.json, review.json bridge, patched repo, report markdown, etc.).\\n- Record whether Dockerfile.artifact_only and Dockerfile.artifact_baseline exist.\\n- Record whether the verifier already emits validation_result-style structured output.\\n\\nWhy\\nCurrent support is uneven across SDLC suites, and we need a canonical source of truth before standardizing contracts or closing gaps.\",\"acceptance_criteria\":\"1. All 275 canonical tasks are classified by evaluator family, expected output artifact(s), and current artifact-mode support. 2. The audit identifies exact tasks missing artifact_only or answer.json bridge support. 3. Results are stored in repo-visible metadata or a generated audit artifact that can drive follow-on work.\",\"status\":\"open\",\"priority\":1,\"issue_type\":\"task\",\"owner\":\"sjarmak@users.noreply.github.com\",\"created_at\":\"2026-03-09T16:05:19Z\",\"created_by\":\"sjarmak\",\"updated_at\":\"2026-03-09T16:05:19Z\"}"} |
0 commit comments