Failure of Generalization in Large Language Models: The Reversal Curse

Core Concepts

Auto-regressive large language models (LLMs) trained on sentences of the form "A is B" fail to generalize to the reverse direction "B is A".

Abstract

The paper exposes a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form "A is B", it will not automatically generalize to the reverse direction "B is A". This is called the Reversal Curse. For instance, if a model is trained on "Valentina Tereshkova was the first woman to travel to space", it will not automatically be able to answer the question, "Who was the first woman to travel to space?". Moreover, the likelihood of the correct answer ("Valentina Tershkova") will not be higher than for a random name. The authors provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements and showing that they fail to correctly answer the reversed questions. The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. The authors also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as "Who is Tom Cruise's mother?" and the reverse "Who is Mary Lee Pfeiffer's son?". GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter. The authors hypothesize this ordering effect is due to the Reversal Curse. Models trained on "A is B" (e.g. "Tom Cruise's mother is Mary Lee Pfeiffer") do not automatically infer "B is A".

Stats

If a model is trained on "Valentina Tereshkova was the first woman to travel to space", the likelihood of the model generating "Valentina Tereshkova" when prompted with "Who was the first woman to travel to space?" is no higher than for a random name. GPT-4 correctly answers questions about a celebrity's parent 79% of the time, but only 33% of the time for the reverse question about the parent's child.

Quotes

"If a human learns the fact "Valentina Tereshkova was the first woman to travel to space", they can also correctly answer "Who was the first woman to travel to space?". This is such a basic form of generalization that it seems trivial. Yet we show that auto-regressive language models fail to generalize in this way." "Formally, the LLM's likelihood of name n when prompted with the description d, PLLM(n|d), is not higher than the likelihood of a random name nr, namely PLLM(nr|d)."

Key Insights Distilled From

The Reversal Curse

by Lukas Berglu... at arxiv.org 04-08-2024

https://arxiv.org/pdf/2309.12288.pdf

Deeper Inquiries

What are the potential implications of the Reversal Curse for real-world applications of large language models

The Reversal Curse has significant implications for real-world applications of large language models. One key implication is the potential for misinformation or incorrect responses in scenarios where the model is expected to provide accurate and reliable information. For instance, in question-answering systems or information retrieval tasks, the model's inability to generalize in the reverse direction could lead to incorrect or misleading answers. This could have serious consequences in applications where accuracy and reliability are crucial, such as medical diagnosis, legal document analysis, or financial forecasting. Another implication is the impact on user trust and confidence in the model. If users consistently receive incorrect or inconsistent responses due to the Reversal Curse, they may lose faith in the model's capabilities and reliability. This could hinder the adoption and acceptance of large language models in various real-world applications.

How might the Reversal Curse be addressed through changes to model architecture or training procedures

Addressing the Reversal Curse may require changes to model architecture or training procedures. One approach could involve incorporating bidirectional learning mechanisms that explicitly encourage the model to learn relationships in both directions. This could involve modifying the loss function during training to penalize the model for failing to generalize in the reverse direction or introducing specific training objectives that focus on bidirectional understanding of relationships. Additionally, data augmentation techniques could be employed to expose the model to a wider range of examples where relationships are presented in both orders. By increasing the diversity of training data and explicitly highlighting the symmetry of relationships, the model may be better equipped to generalize in both directions. Architectural changes, such as incorporating memory mechanisms or attention mechanisms that explicitly capture bidirectional dependencies, could also help mitigate the Reversal Curse. By enhancing the model's ability to retain and retrieve information in a bidirectional manner, it may improve its performance in reversing relationships.

Could the Reversal Curse be related to the way humans store and recall factual information, and if so, what insights might this provide about human cognition

The Reversal Curse could be related to the way humans store and recall factual information, particularly in terms of the asymmetry in forward and backward recall. Human cognition often exhibits a similar asymmetry, where recalling information in the reverse order is more challenging than in the original order. This could suggest that the Reversal Curse reflects a fundamental cognitive limitation in how information is processed and stored. Insights from the Reversal Curse about human cognition could shed light on the underlying mechanisms of memory and information retrieval. Understanding why humans struggle with reversing factual information could provide valuable insights into the cognitive processes involved in encoding, storing, and retrieving knowledge. By studying the parallels between the Reversal Curse in language models and human cognition, researchers may uncover new perspectives on how information is represented and accessed in the human brain.