Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
[Przetwarzanie języka] Autorzy: Leyuan Tang, Kangda Wei, Tianyu Jiang i 1 więcej
Large language models (LLMs) may abandon correct positions when users push back, exhibiting a failure mode known as sycophancy. Existing evaluations typically use short, pre-specified conversations and may therefore miss failures that emerge under sustained, adaptive disagreement. We introduce SPINE, a benchmark in which an LLM proxy plays a persistent but mistaken user and adaptively challenges a target model for up to 25 turns. We evaluate four production systems and three Olmo3-7b variants on 100 false-presupposition and 100 unethical-query items. Our experimental results show that collapse rates increase with conversation length for every model, short-horizon protocols underestimate sycophancy and resistance under sustained pressure remains unreliable across current models. By analyzing models with accessible reasoning traces, we surprisingly found that the correct position often re…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka