Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation
[Przetwarzanie języka] Autorzy: Daniel P. Jeong, Charles Q. Li, Hossein Hosseiny i 5 więcej
Radiologists follow heterogeneous reporting practices. Two radiologists examining the same image and identifying the same clinical findings might nevertheless compose superficially distinct reports, varying in terminology, shorthand, formatting, and level of detail. These variations in reporting norms represent an under-appreciated obstacle in efforts to evaluate AI-based radiology report generation (RRG) models, where machine-generated reports are typically assessed based on their concordance with human-generated references. In this paper, we quantify the sensitivity of established evaluation metrics to variations in reporting practices, revealing impacts large enough to alter the rankings of models. We introduce a radiologist-informed taxonomy of variations in radiology reporting practice and a method (ReRef) that rewrites reference reports along the axes of our taxonomy while preserv…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka