Instruction-Tuned Language Models Exhibit Emergent Cognitive Biases

Core Concepts

Instruction tuning and reinforcement learning from human feedback can introduce or amplify cognitive biases, such as the decoy effect, certainty effect, and belief bias, in large language models.

Abstract

This paper investigates the impact of instruction tuning (IT) and reinforcement learning from human feedback (RLHF) on decision-making and reasoning in large language models (LMs). The authors focus on three well-established cognitive biases: the decoy effect, the certainty effect, and the belief bias. The authors create an experimental dataset using semi-automatically generated decision tasks for each bias. They then evaluate the degree of bias exhibited by several pretrained LMs and compare them to their corresponding fine-tuned variants. The key findings are: Models fine-tuned using IT and RLHF show higher levels of bias compared to their pretrained counterparts. This suggests that the fine-tuning process, intended to enhance model performance, inadvertently introduces biases into the decision-making process. The biases observed in the models align with the well-established theory on irrational biases inherent in human decision-making processes, highlighting the potential connection between human biases and the biases induced by tuning methods. IT alone can amplify biases, as evident from the comparison between T5 and Flan-T5, as well as Mistral and Mistral-Instruct models. RLHF also contributes to bias amplification, as seen in the comparison between DaVinci-002 and DaVinci-003. The larger Flan-T5-XXL model exhibits higher bias scores in some cases, suggesting that model size can also influence the emergence of biases. The authors conclude that the presence of these biases in instruction-tuned LMs highlights an important limitation of tuning based on instructions or human feedback, and emphasizes the need for further research to understand and mitigate cognitive biases in language models.

Stats

"Instruction tuning (IT) and reinforcement learning from human feedback (RLHF) improve the abilities of large language models (LMs) dramatically." "Our findings highlight the presence of these biases in various models from the GPT-3, Mistral, and T5 families." "Notably, we find a stronger presence of biases in models that have undergone instruction tuning, such as Flan-T5, Mistral-Instruct, GPT3.5, and GPT4."

Quotes

"Recent studies show that instruction tuning (IT) and reinforcement learning from human feedback (RLHF) improve the abilities of large language models (LMs) dramatically." "Our findings highlight the presence of these biases in various models from the GPT-3, Mistral, and T5 families." "Notably, we find a stronger presence of biases in models that have undergone instruction tuning, such as Flan-T5, Mistral-Instruct, GPT3.5, and GPT4."

Key Insights Distilled From

Instructed to Bias

by Itay Itzhak,... at arxiv.org 04-02-2024

https://arxiv.org/pdf/2308.00225.pdf

Deeper Inquiries

What are the potential real-world implications of the identified cognitive biases in instruction-tuned language models, and how can they be mitigated

The identified cognitive biases in instruction-tuned language models can have significant real-world implications, especially in decision-making tasks where these models are utilized. For instance, biases like the decoy effect and the certainty effect can lead to suboptimal decision outcomes, affecting the quality and reliability of the model's recommendations. To mitigate these biases, several strategies can be employed. One approach is to diversify the training data used for fine-tuning, incorporating a wide range of scenarios to reduce the model's susceptibility to specific biases. Additionally, implementing bias detection algorithms during model training and deployment can help flag instances where biases are likely to influence the model's decisions, allowing for corrective actions to be taken. Regular audits and evaluations of the model's performance in real-world applications can also help identify and address biases as they arise, ensuring the model's outputs remain unbiased and reliable.

How do the sources of these biases (pretraining vs. fine-tuning) differ, and what are the underlying mechanisms that lead to their emergence

The sources of biases in instruction-tuned language models can stem from both the pretraining data and the fine-tuning process. During pretraining, models may inadvertently learn biases present in the data they are trained on, which can be further amplified or altered during the fine-tuning stage. Pretraining biases are often a result of the data used to train the model, reflecting societal biases, stereotypes, or imbalances present in the training data. On the other hand, fine-tuning biases can arise from the specific instructions or human feedback provided to the model during the fine-tuning process, shaping the model's decision-making patterns. The underlying mechanisms that lead to the emergence of biases in these models are complex and multifaceted, influenced by the nature of the training data, the fine-tuning objectives, and the model architecture. Understanding these mechanisms requires a detailed analysis of the training pipeline, data sources, and model behavior at different stages of training.

What other types of biases, beyond the cognitive biases examined in this study, might be present in instruction-tuned language models, and how can they be systematically investigated

In addition to the cognitive biases examined in this study, instruction-tuned language models may exhibit various other types of biases that can impact their decision-making capabilities. Some potential biases include confirmation bias, availability bias, anchoring bias, and social biases related to gender, race, or cultural stereotypes. To systematically investigate these biases, researchers can design specific experiments and datasets tailored to each type of bias, similar to the approach taken in studying cognitive biases. By creating controlled scenarios that elicit biased responses from the models, researchers can quantify the presence and magnitude of these biases and develop strategies to mitigate them. Additionally, leveraging diverse datasets, incorporating bias detection algorithms, and conducting thorough model evaluations can help uncover and address a wide range of biases in instruction-tuned language models.