DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?
[Inżynieria oprogramowania] Autorzy: Yuxuan Jiang, Chenglin Liu, John Rose i 6 więcej
LLM agents show strong potential in software engineering tasks such as competitive programming and code synthesis. Existing benchmarks, however, largely focus on small-scale settings such as self-contained Python modules, leaving distributed systems, an essential class of production software, under-examined. We introduce DistDebug-Bench, a benchmark for evaluating agents' ability to diagnose complex-system failures at the code level. It curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns. Through an extensive evaluation of state-of-the-art LLMs and agents, we identify substantial gaps between current agents and real-world debugging tasks. A recurring gap is tunnel vision due to a lack of explicit reasoning structure and nondeterminism from inconsistent tool-use scope. The benchmark and evaluator are a…
- Kategoria arxiv
- cs.SE
- Kategoria
- Inżynieria oprogramowania