SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents
[Przetwarzanie języka] Autorzy: Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov i 1 więcej
The rapid proliferation of conversational AI agents necessitates evaluation methods that assess performance across diverse dialogue scenarios, utilizing multiple situational criteria, as appropriate behavior depends on context and encompasses various aspects. Existing dialogue benchmarks typically use static natural conversations, which underrepresent rare situations---a critical safety gap, as low-probability events become frequent failures at production scale. However, limited sample sizes force a choice: either maintain natural dialogues for realism or turn to adaptive situation sampling for comprehensive yet statistically reliable coverage. We introduce SIT-Eval, an adjustable framework for evaluating situation-based conversational agents. We formalize an agent's desired behavior as situational rules: if situation is P, follow rules Q. At each conversation turn, an LLM-interlocutor …
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka