Coding tests found no clear overall winner between setups
A coding harness is the set of tools, instructions, and rules around its model. This study found no clear overall winner between the setups it compared. For your own setup, check whether the code works, whether the system finished, and what the run cost. Keep those checks separate.
What the source reports
The study tested coding models with different harnesses, the tools, instructions, and rules that guide their work. It found no clear overall advantage for either setup in its main comparisons. That does not prove the setups are equal. Getting the code right was also different from finishing a run. Of 81 runs stopped by a time limit, 22 had already produced a code change that passed the test. Cost was harder to pin down. Some runs had no usage record, so the author could not settle which setup cost less on the Anthropic bill. The paper corrected earlier cost figures after finding an error in its own usage tracking.
Original source
- Title
- Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite
- Author
- Mohsen Arjmandi
- Publication
- arXiv
- Date
- Tuesday, September 8, 2026