Core Concepts
Large language models have significantly advanced text understanding and generation, but there is a debate on whether these models are consistent with human and societal values and norms. This paper introduces a new challenge to test whether LLMs understand fundamental social norms, and proposes a multi-agent framework to improve their performance.
Abstract
The paper presents a new dataset called "Social" to examine the ability of large language models (LLMs) to understand human social norms. The dataset consists of 12,383 high-quality multi-choice questions belonging to 402 skills, covering a wide range of social norms including rules, laws, culture, history, and communication.
The authors evaluate the performance of state-of-the-art LLMs, including GPT3.5-Turbo and LLaMA2-Chat, on the Social dataset. The results show that recent advancements in LLMs, particularly the use of reinforcement learning with human feedback (RLHF), have significantly improved the models' ability to understand social norms. However, the best-performing LLMs are still slightly below the performance of average elementary students.
To further enhance LLMs' understanding of social norms, the authors propose a multi-agent framework called "SocialAgent". SocialAgent integrates three LLM agents: a retrieval agent to collect relevant web knowledge, a programming agent to perform symbolic reasoning, and a reasoning agent to trigger step-by-step logical thinking. The ensemble of these agents helps LLMs reach parity with human performance on the Social dataset.
The paper also provides a detailed analysis of the dataset, including the skill distribution, grade-level performance, and case studies. The findings suggest that while LLMs have made progress in understanding fundamental social norms, there is still significant room for improvement, especially in more advanced social norm skills.
Stats
The Social dataset contains 12,383 multi-choice questions covering 402 unique skills across two subjects: social studies and language arts.
Quotes
"Social norms are social and shared among members of a group. It includes topics representing socially acceptable ways of living by a group of people in a society, such as rules, laws, culture, history, and communication."
"Understanding these skills is important to the wide adoption of LLMs."
"SocialAgent helps improve the zero-shot performance in understanding social norms."