LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
[Przetwarzanie języka] Autorzy: Sebastian Fox, Luke Markham, Ryan Lail i 1 więcej
Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation …
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka