A Living Benchmark for Information Retrieval from Electronic Health Records
[Sztuczna inteligencja] Autorzy: Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani i 23 więcej
Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how clinicians retrieve and synthesize information from patient records. Their safety and utility depend on rigorous evaluation, yet existing benchmarks are manually curated, costly to update, and rapidly become obsolete with evolving technological advancements. We present a scalable framework that automatically generates question--answer pairs from longitudinal EHR notes. Nineteen clinicians validate the benchmark generator, producing the Benchmark for Retrieving Information in EHRs (BRIE), a continuously maintainable evaluation dataset. Across nine LLMs and five inference strategies, state-of-the-art systems frequently omit clinically important information, particularly for questions requiring synthesis across multiple documents and encounter…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja