SWE-Bench Pro
Real repository issues resolved end-to-end, graded by hidden tests. Vetta, Codex, Claude Code, Hermes, and Stock Pi each ran the same fifteen-task slice with GLM-5.3-Flash.
Cost per successful task · same model and task slice
$0.0398 / success · 13% less than Codex
Vetta solved ten of fifteen tasks at $0.0398 per successful task. Codex solved eight at $0.0460 each. Hermes solved twelve at up to $0.2510 each. Claude Code solved all fifteen and Stock Pi solved nine. Their dollars are not shown, because their recorded charges are higher than their token counts allow on this rate card.
Efficiency leaderboard?Five matched harnesses on one model and task slice; click a column to re-sort.
Every row uses the same model, provider route, task slice, and one-attempt protocol. The harness determines how much context, time, and model spend each attempted task consumes.
Sorted by Cost / successful task ↑
| # | Harness | Model | |||||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | Vetta | GLM-5.3-Flash ◆ | $0.0398 | 10 / 15 | 66.7% | 10.1 min | 811.7k | 8.6k | 38.9 |
| 2 | Codex | GLM-5.3-Flash | $0.0460 | 8 / 15 | 53.3% | 6.6 min | 1.3M | 9.0k | — |
| 3 | Hermes | GLM-5.3-Flash ◆ | $0.2510 | 12 / 15 | 80.0% | 32.6 min | 2.6M | 13.7k | — |
| 4 | Claude Code | GLM-5.3-Flash | — | 15 / 15 | 100.0% | 15.9 min | 4.0M | 26.1k | — |
| 5 | Pi | GLM-5.3-Flash | — | 9 / 15 | 60.0% | 19.7 min | 13.6M | 22.9k | — |
◆ on the cost-efficiency frontier — no other configuration both solves more tasks and costs less per successful task. Every row uses GLM-5.3-Flash on the same task slice.
Why we run this benchmark
Issue resolution is the workload most teams actually buy an agent for. Holding the model and tasks fixed isolates how much completion rate, latency, and spend come from the machinery around the model, with Vetta benchmarked as another row against public alternatives.
Methodology
Tasks come from SWE-Bench Pro, pinned to a fixed dataset commit. Each attempt starts from a clean containerized checkout of the repository at the issue commit and runs unattended; grading is the suite's hidden tests — binary and automatic, no partial credit, no human judging.
Every harness ran the identical fifteen-task slice with GLM-5.3-Flash through OpenRouter. Invalid infrastructure attempts were repaired and rerun rather than scored as task failures. The published field contains 75 usable trials.
For Vetta and Codex, cost is the model API charge recorded by each run. Hermes did not record dollars, so its tokens are priced on the shared GLM-5.3-Flash rate card with no cache discount. That makes its cost an upper bound. Claude Code and Stock Pi recorded charges above the most their tokens could cost on that card, so we do not show them.
These figures come from our internal benchmark database. The trial-level rows are not public yet.
Want to try it? Deploy Vetta or explore the CLI.