AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions
[Przetwarzanie języka] Autorzy: Alexey A. Bokov, Ivan I. Panin, Nikolay B. Nechaev i 10 więcej
Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clinically coherent, actionable, and safe. We developed a multidimensional benchmark using 300 diagnostically resolved NEJM--MedQA cases: 110 for physician reference ratings and Judge calibration and 190 for held-out evaluation. Eight single-model pipelines generated independent role-specific opinions and interactive-consilium reports under free-form and structured clinical-reasoning conditions. A physician-calibrated LLM Judge evaluated 3,040 final reports and the corresponding structured primary-role outputs. Model identity separated report quality more strongly than response format. Structured final reports had higher overall mean scores in seven of eight model pipelines, with their clearest advantage in management (2.740 vs. 2.436). Compare…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka