Puls Dane publiczne, aktualizowane automatycznie
arXiv – nowe publikacje publikacja

Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget Constraints

[Inżynieria oprogramowania] Autorzy: Srinath Srinivasan, Tim Menzies

Teams that adopt LLM agents for issue resolution must decide which agent to pay for, and how much to spend finding out. Leaderboard ranks give weak evidence for this decision. Accuracy differences may be statistically unresolved, reported costs may be inconsistent, and evaluating candidate agents can itself be expensive. We investigate budget-constrained agent selection using 58,197 run--task records extracted from the Holistic Agent Leaderboard (HAL), focusing on SWE-bench and six additional benchmarks. Across seven benchmarks, 27 of 35 steps along the accuracy--cost Pareto frontiers are statistically unresolved. On SWE-bench, a run costing $34.16 trails the $170.78 leader by 14.6 percentage points, but the available tasks cannot statistically distinguish their accuracy. Reported costs also differ from standardized token-based estimates by up to 11.7x, while selective task evaluation c…

Kategoria arxiv
cs.SE
Kategoria
Inżynieria oprogramowania
Pdf
https://arxiv.org/pdf/2611.21515v1

Otwórz oryginał w źródle ↗

Więcej z tego źródła

Wszystkie