Interpreting Large Language Model Behavior Using Shapley Value Analysis: Uncovering the Impact of Token Noise on Decision-Making

Core Concepts

Large language models (LLMs) exhibit decision-making patterns that can diverge significantly from human cognition, raising concerns about their validity as proxies for human subjects in research. This paper presents a novel Shapley value-based approach to quantify the relative contribution of each prompt component in shaping LLM outputs, revealing the outsized impact of "token noise" - tokens with minimal informative content - on LLM decisions. The findings underscore the need for a more nuanced understanding of LLM behavior and caution against over-relying on these models as substitutes for human subjects.

Abstract

The paper introduces a novel approach based on Shapley values from cooperative game theory to interpret the behavior of large language models (LLMs) and quantify the relative contribution of each prompt component to the model's output.
The key highlights are:

LLMs have shown the potential to exhibit reasoning patterns akin to humans, making them appealing as proxies for human subjects in various domains like marketing research. However, there are concerns about the validity of using LLMs in this capacity due to glaring divergences from human cognition and the sensitivity of LLM responses to prompt variations.

The Shapley value method treats the prompt elements as "players" in a game and quantifies their relative contribution to the LLM's decisions. This allows the identification of "token noise" - tokens with minimal informative content that exert outsized influence on the model's choices.

Two applications are presented:
a. A discrete choice experiment reveals that LLM decisions are heavily influenced by low-information tokens, casting doubt on the validity of using LLMs as proxies for human subjects in marketing research.
b. An investigation of the framing effect in LLMs shows that the apparent sensitivity to framing is largely an artifact of token noise, and prompt optimization can mitigate this effect.

The findings underscore the need for a more nuanced understanding of LLM behavior and caution against over-relying on these models as substitutes for human subjects in research settings. Researchers are encouraged to report results conditioned on specific prompt templates and exercise care when drawing parallels between human behavior and LLMs.

Stats

The following sentences contain key metrics or important figures used to support the author's key logics:
The Shapley value of player 𝑥𝑖 is the average of player 𝑥𝑖's contribution to each coalition 𝒮 weighted by |𝒮|! (𝑁−|𝒮| −1)!, the number of permutations in which the coalition can be formed.
The unnormalized Shapley value for "only" stands at −0.08, implying that the presence of this word increases the probability of choosing flight "B" by 8%.
The cosine similarity between the Shapley value distributions with and without positive framing for Llama-7B-Chat is ≃0.90, which suggests that the overall (noisy) decision-making process of the LLM remains largely consistent across different framings.

Quotes

"The fact that the highest Shapley values belong to {"flights", "Flight [A]", "Flight [B]"} indicates that the model's decision—choosing flight "A" or "B"—is mostly swayed by words that do not actually provide any details about the flight options."
"Without the Shapley value analysis, identifying the key tokens responsible for the apparent framing effect and devising an effective strategy to mitigate it would be a challenging and arduous task, especially with longer prompts."

Key Insights Distilled From

Wait, It's All Token Noise? Always Has Been

by Behnam Moham... at arxiv.org 04-03-2024

https://arxiv.org/pdf/2404.01332.pdf

Deeper Inquiries

How can the Shapley value method be extended to analyze the behavior of LLMs in more complex, multi-step decision-making tasks?

The Shapley value method can be extended to analyze the behavior of LLMs in more complex, multi-step decision-making tasks by considering the sequential nature of the decisions. In multi-step tasks, each decision point can be treated as a separate prompt, and the Shapley values can be calculated for each decision point to understand the contribution of different tokens at each step. By breaking down the decision-making process into smaller components, the Shapley values can provide insights into how the model's choices evolve over multiple steps. Additionally, incorporating feedback loops or reinforcement learning mechanisms into the analysis can help capture the dynamic nature of decision-making in LLMs. This extension would allow for a more detailed understanding of how LLMs process information and make decisions in complex scenarios.

To what extent do the token noise effects observed in this study generalize to other types of LLM applications beyond marketing and consumer behavior research?

The token noise effects observed in this study are likely to generalize to other types of LLM applications beyond marketing and consumer behavior research to some extent. Token noise, where certain tokens have a disproportionate influence on the model's decisions despite carrying minimal semantic information, can impact the reliability and robustness of LLM outputs in various applications. In tasks where LLMs are required to make decisions based on textual inputs, the presence of token noise can lead to biased or unreliable results. For instance, in natural language processing tasks such as text generation or sentiment analysis, token noise could affect the quality and accuracy of the model's outputs. Therefore, it is essential for researchers and practitioners in diverse fields utilizing LLMs to be aware of the potential influence of token noise and take steps to mitigate its effects to ensure the validity of their results.

What other complementary techniques, beyond Shapley values, could provide a more comprehensive understanding of the factors driving LLM decision-making and their relationship to human cognition?

In addition to Shapley values, several complementary techniques could provide a more comprehensive understanding of the factors driving LLM decision-making and their relationship to human cognition. One such technique is Layer-wise Relevance Propagation (LRP), which attributes the model's output to individual neurons or layers, offering insights into the internal workings of the LLM. Another approach is Integrated Gradients, which calculates the integral of the gradients along the path from a baseline input to the actual input, highlighting the importance of different features in the decision process. Furthermore, adversarial testing can be employed to evaluate the robustness of LLMs by introducing perturbations to the input and observing the model's response. Additionally, cognitive psychology experiments conducted on human subjects can provide valuable insights into decision-making processes, which can be compared with LLM behavior to identify similarities and differences. By combining these techniques with Shapley values, researchers can gain a more holistic understanding of LLM decision-making and its alignment with human cognition.