Solving Lateral Thinking Puzzles with Large Language Models: An Evaluation of the Connections Puzzle

Core Concepts

Large language models can solve a substantial proportion of Connections puzzles, but struggle with categories requiring abstract or lateral thinking.

Abstract

The Connections puzzle, published daily by the New York Times, tasks players with dividing a grid of 16 words into 4 groups of 4 related words. Solving the puzzle requires both common linguistic knowledge and abstract reasoning, as the categories increase in complexity from "simple" to "tricky".
The authors investigate the ability of sentence embedding baselines and large language models (LLMs) in the GPT family to solve Connections puzzles. They find that the best-performing sentence embedding model (MPNET) has an 11.6% success rate, while GPT-3.5-TURBO and GPT-4-TURBO achieve 6.43% and 29.2% success rates respectively.
The authors observe that the LLMs struggle particularly with categories involving non-semantic properties of words, abstract features, or usage in context. They also find that the LLMs' performance is highly dependent on whether their initial guess is correct, with a substantial drop in success rate if the first guess is incorrect or nearly correct.
The authors further examine the impact of chain-of-thought prompting on GPT-4-TURBO, finding a significant boost in performance from 29.2% to 38.93% average success rate. They also evaluate a more challenging variant of the Connections puzzle where all 4 groups must be submitted simultaneously, observing mixed results across models.
Overall, the authors conclude that the Connections puzzle presents a fertile ground for studying the capabilities and limitations of modern language models in encoding and retrieving semantic information, and propose it as a useful benchmark for evaluating abstract reasoning in NLP systems.

Stats

The best-performing sentence embedding baseline (MPNET) solves 11.6% of Connections puzzles on average.
GPT-3.5-TURBO achieves a 6.43% average success rate on the Connections puzzle.
GPT-4-TURBO achieves a 29.2% average success rate on the Connections puzzle.
Chain-of-thought prompting boosts GPT-4-TURBO's performance to 38.93% average success rate.

Quotes

"The Connections puzzle acts as both a test of linguistic understanding and abstract reasoning."
"We find that the LLMs are often stumped by categories which involve non-semantic properties of words, abstract features, or usage in context."
"Studying the ways in which LLM and human player behaviors differ could shed light on the differences in their underlying representations of words and meanings."

Key Insights Distilled From

Missed Connections: Lateral Thinking Puzzles for Large Language Models

by Graham Todd,... at arxiv.org 04-19-2024

https://arxiv.org/pdf/2404.11730.pdf

Deeper Inquiries

How could incorporating external knowledge bases or multimodal information improve language models' performance on the Connections puzzle?

Incorporating external knowledge bases or multimodal information could significantly enhance language models' performance on the Connections puzzle by providing additional context and semantic understanding. External knowledge bases like WordNet could offer a wealth of information on word relationships, synonyms, and common associations, which could help language models make more accurate connections between words in the puzzle. By accessing structured knowledge sources, language models can expand their understanding beyond the text data they were trained on, enabling them to make more nuanced and informed decisions when categorizing words in the puzzle.
Multimodal information, such as combining text with images or audio data, could also benefit language models in solving the Connections puzzle. For instance, visual cues or context from images could provide additional clues or associations between words that are not apparent from text alone. By integrating different modalities of information, language models can develop a more comprehensive understanding of the puzzle and make more sophisticated connections between words based on a diverse range of inputs.
Overall, incorporating external knowledge bases and multimodal information can enrich language models' knowledge representation, improve their ability to make abstract connections, and enhance their performance on complex linguistic tasks like the Connections puzzle.

What are the key differences in the strategies and reasoning processes used by human players versus language models when solving Connections puzzles?

Human players and language models employ distinct strategies and reasoning processes when solving Connections puzzles, reflecting differences in cognitive capabilities and information processing.

Semantic Understanding: Human players rely on their comprehensive semantic understanding of words, drawing on real-world knowledge, experiences, and common associations to categorize words in the puzzle. In contrast, language models leverage statistical patterns and embeddings learned from vast text corpora to make connections between words based on co-occurrence and contextual information.

Lateral Thinking: Human players often engage in lateral thinking, considering unconventional or creative associations between words to solve challenging categories in the puzzle. Language models, on the other hand, may struggle with abstract or lateral connections that require intuitive leaps or non-linear reasoning, as their training data primarily consists of structured linguistic patterns.

Adaptability: Human players can adapt their strategies based on feedback, trial and error, and intuitive insights during the puzzle-solving process. In comparison, language models follow predefined algorithms and prompts, limiting their ability to dynamically adjust their reasoning based on evolving information or feedback from the puzzle.

Contextual Understanding: Human players can infer contextual nuances, word connotations, and subtle meanings that may not be explicitly encoded in the text. Language models, while proficient in capturing surface-level semantic information, may struggle with understanding context-dependent word relationships or abstract concepts that require deeper comprehension.

Overall, human players excel in leveraging creativity, intuition, and contextual understanding to solve Connections puzzles, while language models demonstrate proficiency in pattern recognition, statistical inference, and leveraging large-scale linguistic data to make connections between words.

Could the Connections puzzle be used as a testbed for developing language models with stronger capabilities in analogical and lateral thinking?

Yes, the Connections puzzle holds great potential as a testbed for developing language models with enhanced capabilities in analogical and lateral thinking. By challenging language models to identify abstract relationships, unconventional associations, and creative connections between words, the puzzle can serve as a valuable benchmark for evaluating and improving the models' reasoning abilities.

Analogical Reasoning: The Connections puzzle requires identifying similarities and commonalities between words, akin to analogical reasoning tasks. By training language models on a diverse set of Connections puzzles, developers can enhance the models' ability to recognize and apply analogies, fostering a deeper understanding of semantic relationships and conceptual mappings.

Lateral Thinking: The puzzle's emphasis on lateral thinking encourages language models to explore unconventional connections and think outside the box when categorizing words. By exposing models to a wide range of challenging categories that require creative interpretations, developers can stimulate lateral thinking skills and promote innovative problem-solving approaches in the models.

Creative Problem-Solving: Incorporating the Connections puzzle into language model training can foster creativity and innovation in the models' reasoning processes. By encouraging models to make unexpected connections and consider diverse perspectives, the puzzle can push the boundaries of traditional linguistic tasks and inspire novel solutions to complex problems.

In conclusion, leveraging the Connections puzzle as a testbed for analogical and lateral thinking can enhance language models' cognitive flexibility, creativity, and problem-solving capabilities, paving the way for more advanced and human-like reasoning in artificial intelligence systems.