Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models
[Sztuczna inteligencja] Autorzy: Junru Zhu, Shiming Xie, Aime Lu Fan Chen i 4 więcej
Tool-using agents can fail twice: a required tool can fail, and the agent can then report success without the evidence needed to justify it. Existing benchmarks often entangle this reporting failure with tool selection, recovery, and environment dynamics. We introduce Failure-Transparent Agents (FTA), a controlled benchmark that fixes the failed observation and required evidence state before generation, making post-failure claims directly auditable. FTA contains 100 tasks with deterministic failure traces spanning five failure families, a neutral control, and four user-pressure conditions, and evaluates unsupported claims alongside useful recovery. Across six models, three response policies, and 3,600 human-annotated responses, false-success rates are 22.8% under the baseline policy, 9.3% with a transparency instruction, and 0.8% with a structured evidence contract. Fabricated-detail ra…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja