[Paper Review] Self-Attentive Associative Memory

Hung Lê, Truyen Tran+1 more|arXiv (Cornell University)|Feb 9, 2020

Topic Modeling6 citations

TL;DR

This paper proposes Self-Attentive Associative Memory (SAM), a novel neural network module that separates item memory from relational memory using outer products to model high-order relationships between memory elements. The method enables end-to-end learning for both memorization and relational reasoning, achieving competitive performance across diverse tasks including geometry, graph learning, and question answering.

ABSTRACT

Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions. A rich representation of relationships between memory pieces urges a high-order and segregated relational memory. In this paper, we propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory). The idea is implemented through a novel Self-attentive Associative Memory (SAM) operator. Found upon outer product, SAM forms a set of associative memories that represent the hypothetical high-order relationships between arbitrary pairs of memory elements, through which a relational memory is constructed from an item memory. The two memories are wired into a single sequential model capable of both memorization and relational reasoning. We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks, from challenging synthetic problems to practical testbeds such as geometry, graph, reinforcement learning, and question answering.

Motivation & Objective

• To overcome the limitations of existing neural networks with external memory, which rely on single, lossy representations of memory interactions.
• To enable high-order relational reasoning by explicitly modeling relationships between memory elements beyond simple storage.
• To design a unified sequential model that integrates both item memory and relational memory for joint memorization and reasoning.
• To improve performance on complex tasks requiring relational inference, such as question answering and reinforcement learning.
• To demonstrate the effectiveness of the two-memory architecture across diverse machine learning benchmarks.

Proposed method

• The SAM operator constructs relational memory via outer products between pairs of item memory vectors, forming a high-dimensional associative memory matrix.
• Each entry in the matrix represents a hypothetical high-order relationship between two memory elements, enabling rich relational representations.
• The model uses self-attention mechanisms to dynamically attend to relevant relational patterns during inference.
• Item memory stores individual experiences, while relational memory captures the interactions and structural relationships between them.
• The two memory streams are integrated into a single sequential architecture, allowing end-to-end training with both memorization and reasoning objectives.
• The framework supports differentiable learning, enabling gradient-based optimization across both memory types.

Experimental results

Research questions

• RQ1 Can a neural network architecture effectively separate item memory from relational memory to improve reasoning capabilities?
• RQ2 How well can a high-order relational memory, constructed via outer products, capture complex relationships in diverse tasks?
• RQ3 Does integrating both item and relational memory lead to improved performance on tasks requiring relational inference?
• RQ4 Can the proposed SAM operator generalize across synthetic and real-world benchmarks, including graph and reinforcement learning tasks?
• RQ5 How does the SAM model compare to existing memory-augmented networks in terms of memorization and reasoning performance?

Key findings

• The SAM model achieves competitive performance on challenging synthetic tasks requiring relational reasoning, demonstrating its ability to learn complex patterns.
• The model shows strong generalization on practical testbeds, including geometry reasoning, graph-based tasks, and question answering.
• By separating item and relational memory, the architecture enables more expressive and structured representations than single-memory alternatives.
• The use of outer products to form associative memories allows for high-order relationship modeling without explicit supervision on relations.
• The end-to-end trainable design supports effective joint learning of memorization and relational reasoning across diverse domains.
• The method outperforms or matches existing memory-augmented networks on benchmarks such as visual question answering and reinforcement learning environments.