PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios
[Przetwarzanie języka] Autorzy: Ali Zamani, Soroush Esfahanian, Erfan Shahabi i 3 więcej
Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals. Persuasion provides a natural testbed in which both aspects are simultaneously at play, yet prior benchmarks typically evaluate them in isolation using synthetic scenarios. We introduce PerSo, a benchmark that evaluates both aspects on real-world persuasion dialogues from Reddit's ChangeMyView community, using an LLM-in-the-loop pipeline to construct the evaluation tasks, validated through human annotation. Our evaluation of diverse LLMs shows that performance generally scales wit…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka