Disclosure: I maintain Tura. This is a disclosed technical analysis of public benchmark artifacts, not an independent product review.
A question about how we evaluate high-effort coding-agent modes: is one aggregate Pass@1 enough when the mode appears to help very different task shapes differently?
| Task class | High | Max | Delta | Cost multiplier |
|---|---|---|---|---|
| Bounded fixes (DeepSWE, n=7) | 64.3% | 57.1% | -7.1 pp | 2.53x |
| Feature work (DeepSWE, n=95) | 70.2% | 74.6% | +4.4 pp | 2.43x |
| eza Rust-to-Python rewrite (3 harnesses) | 78.8-89.4% | 92.3-94.2% | +4.8-13.5 pp | 2.27-3.27x |
The all-task DeepSWE number is +3.3 pp (69.4% to 72.7%) for 2.42x cost ($3.47 to $8.39 per task). The 7-task fix bucket is far too small to conclude that Max is harmful for fixes; it does show why a global average can be a poor routing rule.
My provisional operational policy is to use High for a bounded failing test with a known change surface, and consider Max when the agent has to discover a rewrite or migration path and validate cross-repository compatibility.
For people building evaluations: what stratification, confidence intervals, or failure taxonomy would you require before treating task shape as an effort-routing signal?
Full methods and tables: Tura Blog — Coding Agents, Benchmarks, and Development
Reproducible artifacts: benchmark/blog_data/eza-replication-gpt56-max-20260717 at main · Tura-AI/benchmark · GitHub