[Paper Review] Superposed Episodic and Semantic Memory via Sparse Distributed Representation

Gerard Rinkus, Jasmin Léveillé|arXiv (Cornell University)|Oct 21, 2017

Domain Adaptation and Few-Shot Learning23 references3 citations

TL;DR

This paper proposes Sparsey, a hierarchical associative memory model that unifies episodic and semantic memory using sparse distributed representations (SDRs). By storing inputs as SDRs in superposition, the model enables single-trial learning where similarity in input space is preserved as intersection in code space, naturally forming a generative semantic model without separate training. Initial results show fast learning (minutes on CPU) on MNIST and video recognition tasks, though not state-of-the-art accuracy.

ABSTRACT

The abilities to perceive, learn, and use generalities, similarities, classes, i.e., semantic memory (SM), is central to cognition. Machine learning (ML), neural network, and AI research has been primarily driven by tasks requiring such abilities. However, another central facet of cognition, single-trial formation of permanent memories of experiences, i.e., episodic memory (EM), has had relatively little focus. Only recently has EM-like functionality been added to Deep Learning (DL) models, e.g., Neural Turing Machine, Memory Networks. However, in these cases: a) EM is implemented as a separate module, which entails substantial data movement (and so, time and power) between the DL net itself and EM; and b) individual items are stored localistically within the EM, precluding realizing the exponential representational efficiency of distributed over localist coding. We describe Sparsey, an unsupervised, hierarchical, spatial/spatiotemporal associative memory model differing fundamentally from mainstream ML models, most crucially, in its use of sparse distributed representations (SDRs), or, cell assemblies, which admits an extremely efficient, single-trial learning algorithm that maps input similarity into code space similarity (measured as intersection). SDRs of individual inputs are stored in superposition and because similarity is preserved, the patterns of intersections over the assigned codes reflect the similarity, i.e., statistical, structure, of all orders, not simply pairwise, over the inputs. Thus, SM, i.e., a generative model, is built as a computationally free side effect of the act of storing episodic memory traces of individual inputs, either spatial patterns or sequences. We report initial results on MNIST and on the Weizmann video event recognition benchmarks. While we have not yet attained SOTA class accuracy, learning takes only minutes on a single CPU.

Motivation & Objective

• To address the lack of focus on episodic memory in deep learning by integrating it with semantic memory in a unified framework.
• To overcome limitations of existing memory modules that use localist coding and require high data movement between networks and memory.
• To leverage sparse distributed representations (SDRs) for efficient, single-trial learning that preserves input similarity in code space.
• To demonstrate that semantic memory emerges naturally as a byproduct of episodic memory storage via superposition.
• To achieve fast, unsupervised learning on standard benchmarks without requiring extensive training.

Proposed method

• The model uses sparse distributed representations (SDRs), or 'cell assemblies', to encode inputs in a way that preserves similarity through intersection in code space.
• Episodic memories are stored as SDRs in superposition within a hierarchical associative memory structure, enabling efficient retrieval and similarity detection.
• The learning rule maps input similarity directly into code space similarity via intersection of SDRs, avoiding the need for backpropagation or separate training.
• The model operates in a hierarchical, spatial/spatiotemporal manner, allowing it to learn both static patterns and temporal sequences.
• Semantic memory is generated as a free side effect of episodic memory storage due to the statistical structure preserved in SDR intersections.
• The system is fully unsupervised and designed for low computational cost, operating efficiently on a single CPU.

Experimental results

Research questions

• RQ1 Can episodic and semantic memory be unified within a single learning framework using sparse distributed representations?
• RQ2 Does superposition of SDRs in memory allow for similarity-preserving representation without explicit training?
• RQ3 Can a single-trial learning algorithm based on SDR intersection achieve effective semantic generalization?
• RQ4 How does the model perform on standard benchmarks like MNIST and video event recognition without backpropagation or GPU acceleration?
• RQ5 To what extent does semantic memory emerge naturally from episodic memory storage via SDR superposition?

Key findings

• The model achieves fast learning—only minutes on a single CPU—on MNIST and Weizmann video event recognition benchmarks.
• Semantic memory emerges as a natural byproduct of episodic memory storage due to the similarity-preserving properties of SDR intersections.
• The model uses sparse distributed representations to store inputs in superposition, enabling efficient, single-trial learning without data movement between modules.
• The system preserves statistical structure of all orders in input data through intersection-based similarity in code space.
• Despite not reaching state-of-the-art accuracy, the model demonstrates a computationally efficient, unsupervised learning mechanism that integrates episodic and semantic memory.