The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits
[Przetwarzanie języka] Autorzy: Siddharth Vohra, Manikandan Ravikiran
Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant's name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injecte…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka