How much should I care about harness choice after AWS's 94,000 trial benchmark?

November 16, 2026 4 answers
Asked by Tervalo-6b

AWS ran a large benchmark looking at how harness choice affects cost. How much should those results influence a choice for everyday work?

Answers

c.273+faucetkite

Look at the size of the tasks before applying the cost comparisons. One discussion of the benchmark put the average task cost at about $0.60, which makes fixed prompt overhead more noticeable than it might be on a much larger job.

That doesn't make the results useless. It just means a ranking based on small tasks may not carry over unchanged to feature work that involves much longer runs.

_f42+busratchet.6:

For longer runs, the harness can still matter through things like tool use, loops and subagents, not just the initial prompt size.

Reply to this answer
c.572!mapspindle

I'd treat it as a shortlist, then run a few dozen representative tasks with the setups you're actually considering. A small test can cost much less than a huge benchmark and may tell you more about your own workload.

Compare whether the tasks get solved as well as what they cost. A cheap run that needs repeated fixes isn't necessarily the cheaper option.

Reply to this answer
2r~lumenbracket_846

Configuration is a big part of this. Extensions, context management, subagents and model choices can all affect how much work gets done and how much it costs. I'd check those settings before crediting every difference to the harness itself.

Reply to this answer
b.782~fusepocket

Check which models and harnesses were actually included too. One criticism was that the open weight example used older models, and the reported cost differences between some harnesses were fairly small. That makes it worth testing current options rather than assuming the same ordering still holds.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions