Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?
[Sztuczna inteligencja] Autorzy: Trizal Garg, Sibayan Mitra, Arya Karanjkar i 6 więcej
Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet evaluation awareness remains almost entirely unstudied in them. We test four instruction-tuned models from three families in five Indic languages alongside English. A set of cues varies whether the model is told that it is monitored and whether the monitoring is a safety test. Models refuse harmful requests more often when they are told that they are being tested. This increase appears in all four models and all six languages. In Indic languages, refusal of harmful requests rises by 14.4 percentage points on averag…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja