The RAT: A Unified Bayesian Model for RAG Evaluation
[Przetwarzanie języka] Autorzy: Pius von Däniken, Felix Matthias Saaro, Mark Cieliebak i 1 więcej
Evaluating Retrieval-Augmented Generation (RAG) systems requires assessing not only end-to-end correctness but also how individual components interact and how errors propagate through the pipeline. We introduce a Bayesian evaluation framework that jointly models retrieval success, abstention behavior, and answer correctness, factorized according to the pipeline's information flow. The model distinguishes task success. Whether the user received a correct answer (from generator success) and whether the generator behaved appropriately given the retrieval outcome. We apply the framework to 27 RAG configurations across three datasets, three retrievers, and three generators, and show that the conditional decomposition reveals substantial behavioral differences between systems that appear equivalent under marginal metrics. We further analyze the annotation allocation problem, demonstrating tha…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka