[Paper Review] Convolutional Neural Associative Memories: Massive Capacity with Noise Tolerance

Amin Karbasi, Amir Hesam Salavati+1 more|arXiv (Cornell University)|Jul 24, 2014

Neural Networks and Applications21 references3 citations

TL;DR

This paper introduces a convolutional neural associative memory framework that achieves exponential storage capacity by exploiting inherent linear redundancy in patterns, enabling noise-tolerant retrieval of massive datasets. By combining convolutional architecture with an iterative learning algorithm that enforces sparsity and subspace constraints, the model corrects a linear fraction of errors and scales efficiently with pattern size.

ABSTRACT

The task of a neural associative memory is to retrieve a set of previously memorized patterns from their noisy versions using a network of neurons. An ideal network should have the ability to 1) learn a set of patterns as they arrive, 2) retrieve the correct patterns from noisy queries, and 3) maximize the pattern retrieval capacity while maintaining the reliability in responding to queries. The majority of work on neural associative memories has focused on designing networks capable of memorizing any set of randomly chosen patterns at the expense of limiting the retrieval capacity. In this paper, we show that if we target memorizing only those patterns that have inherent redundancy (i.e., belong to a subspace), we can obtain all the aforementioned properties. This is in sharp contrast with the previous work that could only improve one or two aspects at the expense of the third. More specifically, we propose framework based on a convolutional neural network along with an iterative algorithm that learns the redundancy among the patterns. The resulting network has a retrieval capacity that is exponential in the size of the network. Moreover, the asymptotic error correction performance of our network is linear in the size of the patterns. We then ex- tend our approach to deal with patterns lie approximately in a subspace. This extension allows us to memorize datasets containing natural patterns (e.g., images). Finally, we report experimental results on both synthetic and real datasets to support our claims.

Motivation & Objective

• To design a neural associative memory with exponentially large storage capacity while maintaining noise tolerance.
• To enable online, streaming learning of patterns as they arrive, suitable for large-scale datasets.
• To extend the framework to handle natural patterns that approximately lie in a low-dimensional subspace.
• To achieve information-theoretically optimal storage and error correction performance for structured patterns.

Proposed method

• Employs a convolutional neural network architecture with shared weights across spatial locations to exploit local pattern redundancy.
• Uses an iterative learning algorithm inspired by Oja and Kohonen (1988), extended with sparsity constraints to enhance noise suppression.
• Imposes subspace constraints by learning a generator matrix G such that all patterns x = G⊤u lie in a low-dimensional subspace.
• Applies belief propagation-style message passing in a bipartite graph (pattern neurons vs. constraint neurons) for error correction during recall.
• Introduces super constraint nodes to model higher-order dependencies and improve error correction performance.
• Extends the model to approximately subspace-lying patterns using robust learning and denoising procedures.

Experimental results

Research questions

• RQ1 Can a neural associative memory achieve exponential storage capacity while correcting a linear fraction of errors?
• RQ2 Can the model learn patterns in a streaming fashion with provable convergence and noise tolerance?
• RQ3 How does the model perform on natural data such as images, which are only approximately subspace-structured?
• RQ4 What is the theoretical limit of storage capacity and error correction in such a framework, and can it be achieved?

Key findings

• The proposed network achieves exponential storage capacity C = υ^(rn), where υ ≥ 2 and 0 < r < 1, making it scalable with network size.
• The model corrects up to a linear fraction of errors, with asymptotic error correction performance linear in the pattern size.
• The iterative learning algorithm successfully learns patterns in a streaming manner, maintaining sparsity and subspace structure.
• Experimental results on synthetic and real datasets confirm high retrieval accuracy and robustness to noise.
• The framework is information-theoretically optimal in terms of storage capacity and error correction, up to constant factors.
• The method acts as a feature extraction tool, learning meaningful low-dimensional representations from structured data.