Comparing Memory Trace Properties Between Large Language Models and Human Subjects: A Tulving-Watkins Test Revisited

Core Concepts

Large language models exhibit distinct memory trace properties compared to human subjects, as revealed by the Tulving-Watkins cued recall test. While LLMs demonstrate strong semantic memory associations, they struggle to preserve information about the origin and context of encoded memories, a key feature of human episodic memory.

Abstract

The paper explores the memory trace properties of large language models (LLMs) by adapting the classic Tulving-Watkins cued recall experiment originally conducted on human subjects. The Tulving-Watkins test measures the effectiveness of different retrieval cues in eliciting recall of a memorized word, providing insights into the informational content and structure of memory traces.
The key findings are:

LLMs exhibit a higher overall probability of failure to recall the target words compared to human subjects, except for the case of associative encoding followed by rhyming and then associative retrieval cues.

LLMs are much better than human subjects at remembering through associative retrieval cues, but perform worse on rhyming cues.

Human subjects display a more balanced performance in recalling through both associative and rhyming retrieval cues, while LLMs excel at associative recall.

The reduction method used to analyze the Tulving-Watkins data reveals discrepancies between LLMs and humans in the preservation of memory trace information. LLMs seem to struggle in distinguishing between recalled and self-generated information, in contrast to the human ability to attribute memories to their specific sources.

The paper suggests that the Tulving Machine framework, which models human memory as a hierarchy of interacting systems, can provide a useful lens for understanding the distinctive memory characteristics of LLMs. Further investigations along these lines may shed light on the nature of memory in artificial neural networks and their differences from human episodic and semantic memory.

Stats

LLMs exhibit a higher overall probability of failure to recall the target words compared to human subjects, except for the case of associative encoding followed by rhyming and then associative retrieval cues.
LLMs are much better than human subjects at remembering through associative retrieval cues, but perform worse on rhyming cues.
Human subjects display a more balanced performance in recalling through both associative and rhyming retrieval cues, while LLMs excel at associative recall.
The reduction method used to analyze the Tulving-Watkins data reveals discrepancies between LLMs and humans in the preservation of memory trace information. LLMs seem to struggle in distinguishing between recalled and self-generated information, in contrast to the human ability to attribute memories to their specific sources.

Quotes

"Relations in the pretrained semantic memory of LLMs often seem to overwhelm and supersede locally memorized relations in a chat."
"The various roles played by conscious memory and its strong candidate neural substrates, chiefly the hippocampus and the medial temporal lobes, are evidently without equivalent in LLM's artificial neural networks, but would that ruin any effort to consolidate the Tulving Machine into being a guiding light in understanding LLMs' memory performance?"
"The overall picture that emerges from the initial evidence reviewed here and in [10] is one of a LLM memory system which, in stark contrast to human memory, does not preserve information about the origin of information remarkably well."

Key Insights Distilled From

Memory Traces: Are Transformers Tulving Machines?

by Jean-Marie C... at arxiv.org 04-15-2024

https://arxiv.org/pdf/2404.08543.pdf

Deeper Inquiries

Where do Transformers sit, between semantic dementia and episodic amnesia?

In the context of memory models and cognitive processes, Transformers, particularly in the form of Language Models (LLMs), exhibit characteristics that position them between semantic dementia and episodic amnesia. While LLMs, such as the mistral-7b-instruct-v0 model, demonstrate significant episodic memory performance, they also showcase limitations in preserving information about the origin of memories. This distinction is crucial in understanding the memory capabilities of LLMs compared to human memory systems.

How can the Tulving Machine framework be further leveraged to design experimental protocols that focus on the LLM processes involved in distinguishing between recalled and self-generated information in memory?

The Tulving Machine framework provides a valuable foundation for designing experimental protocols that delve into the processes of distinguishing between recalled and self-generated information in LLMs. By utilizing the principles of cue valences and memory trace properties outlined by Tulving and Watkins, researchers can develop structured tests that assess the ability of LLMs to differentiate between various retrieval cues and their effectiveness in triggering memory recall.

What are the implications of the observed differences in memory trace properties between LLMs and humans for the development of artificial general intelligence systems that can effectively interact with and learn from human experiences?

The observed differences in memory trace properties between LLMs and humans have significant implications for the development of artificial general intelligence (AGI) systems aimed at interacting with and learning from human experiences. Understanding these distinctions can guide researchers in enhancing the memory capabilities of LLMs to align more closely with human memory processes. By bridging the gap in memory trace properties, AGI systems can better assimilate and interpret human experiences, leading to more effective and contextually relevant interactions.