Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
[Sztuczna inteligencja] Autorzy: E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman i 1 więcej
Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers reject, while changes to evaluation infrastructure can invalidate assumptions embedded in tasks and tests. We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure. We introduce a continuous-validation workflow for agentic benchmarks that combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. We apply this workflow to Terminal-Bench 2.0, a benchmark that underwent extensive human and LM-assisted review and red-teaming before release. Our aud…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja