Independent Test Agent
Does separating an independent Test agent from a Repair agent reduce false-confidence patches without hurting resolution rate?
Hypothesis
A fail-closed harness with independently-trained Test/Repair roles catches more false-confidence patches than a single agent that writes and grades its own tests, at acceptable compute cost.
Background
ExecCritic (arXiv 2609.09133) reports 72.6% on SWE-bench Verified with this split, +11.4 points over a no-test baseline — the strongest reproducible result in the Sept 9 run, with a verified open repository.
Method
Clone the ExecCritic repository; run one small repository task through the fail-closed harness; audit false positives and leaked test validity.
Measurement Plan
Resolved tasks, false-confidence rate, test validity, and compute cost.
Expected Artifact
A fail-closed repo-repair harness, adapted and benchmarked against the paper's own numbers.
Limitations
Reported results depend on isolated sandboxes and substantial compute — reproduction scope will start much smaller.
Status
This experiment is Proposed — queued, not started. Results, data, and measurements will appear here once it actually runs.
