[Paper Review] Aspects of human memory and Large Language Models

Romuald A. Janik|arXiv (Cornell University)|Nov 7, 2023

Topic Modeling4 citations

TL;DR

This paper investigates memory-like behaviors in Large Language Models (LLMs) and finds that they exhibit human-like memory phenomena—such as primacy and recency effects—despite lacking a dedicated memory architecture. These properties emerge from statistical patterns in training data, suggesting that human narrative structures imprint on language models through the way we write, rather than from architectural design alone.

ABSTRACT

Large Language Models (LLMs) are huge artificial neural networks which primarily serve to generate text, but also provide a very sophisticated probabilistic model of language use. Since generating a semantically consistent text requires a form of effective memory, we investigate the memory properties of LLMs and find surprising similarities with key characteristics of human memory. We argue that the human-like memory properties of the Large Language Model do not follow automatically from the LLM architecture but are rather learned from the statistics of the training textual data. These results strongly suggest that the biological features of human memory leave an imprint on the way that we structure our textual narratives.

Motivation & Objective

• To investigate whether Large Language Models (LLMs) exhibit memory-like behaviors similar to human short-term memory.
• To determine whether these memory characteristics arise from architectural design or from statistical regularities in training data.
• To explore the hypothesis that human biological memory shapes the structure of textual narratives, leaving a detectable imprint in language models.
• To compare LLM memory performance with human memory experiments, focusing on serial recall and interference effects.
• To assess whether emergent memory properties in LLMs are due to inductive biases of the transformer architecture or learned from language statistics.

Proposed method

• Conducted serial memory experiments using a list of 20 'has-a' type facts to test recall accuracy in GPT-J, a 6B-parameter LLM.
• Measured recall performance across different positions in the list to detect primacy (early items) and recency (late items) effects.
• Analyzed the impact of semantic interference by introducing conflicting facts about the same entity to assess recall degradation.
• Used positional embeddings in the transformer architecture to examine whether architectural features alone could explain memory effects.
• Compared results from GPT-J with smaller models (e.g., Pythia-70m) to isolate the role of model scale and training regimen.
• Evaluated memory formation time by varying the distance between the query and the target fact in the sequence, observing recall accuracy trends.

Experimental results

Research questions

• RQ1 Do Large Language Models exhibit primacy and recency effects in serial recall tasks, similar to human memory?
• RQ2 Are the observed memory-like behaviors in LLMs due to architectural inductive biases (e.g., positional embeddings) or learned from training data?
• RQ3 How does semantic interference affect recall accuracy in LLMs, and does this mirror human forgetting through interference?
• RQ4 Does the emergence of memory effects depend on model size and training regimen, suggesting learned over architectural properties?
• RQ5 To what extent do the statistical structures of human-written narratives shape the probabilistic language models in LLMs?

Key findings

• GPT-J exhibits a U-shaped recall curve across list positions, demonstrating both primacy and recency effects, mirroring human memory patterns.
• The recency effect is absent in the smaller Pythia-70m model, indicating that the phenomenon emerges only in larger, more complex models.
• Recall accuracy initially increases with distance from the target fact, contradicting architectural bias hypotheses based on positional embeddings.
• Semantic interference significantly reduces recall accuracy when multiple facts about the same entity are present, supporting the role of interference in forgetting.
• The memory-like behaviors in LLMs are not directly induced by the transformer architecture but are learned from statistical regularities in training data.
• The results suggest that human biological memory leaves a detectable statistical imprint on narrative structure, which is then internalized by LLMs during training.