[Paper Review] The Kanerva Machine: A Generative Distributed Memory

Yan Wu, Greg Wayne+2 more|arXiv (Cornell University)|Apr 5, 2018

Generative Adversarial Networks and Image Synthesis19 references13 citations

TL;DR

The Kanerva Machine is a differentiable, generative memory model inspired by Kanerva's sparse distributed memory, using analytically tractable Bayesian updates to enable fast, distributed writing and robust online compression. It outperforms DNCs in training stability, generalization, and capacity, especially on Omniglot and CIFAR-10, by learning adaptive, data-dependent priors through hierarchical variational inference with a memory-augmented VAE framework.

ABSTRACT

We present an end-to-end trained memory system that quickly adapts to new data and generates samples like them. Inspired by Kanerva's sparse distributed memory, it has a robust distributed reading and writing mechanism. The memory is analytically tractable, which enables optimal on-line compression via a Bayesian update-rule. We formulate it as a hierarchical conditional generative model, where memory provides a rich data-dependent prior distribution. Consequently, the top-down memory and bottom-up perception are combined to produce the code representing an observation. Empirically, we demonstrate that the adaptive memory significantly improves generative models trained on both the Omniglot and CIFAR datasets. Compared with the Differentiable Neural Computer (DNC) and its variants, our memory model has greater capacity and is significantly easier to train.

Motivation & Objective

• To address the limitations of existing memory-augmented networks, such as poor generalization and sensitivity to hyperparameters, by designing a memory system that adapts quickly to new data.
• To overcome the inefficiency of slot-based memory (e.g., in DNCs) and the information loss in averaging-based methods (e.g., Neural Statistician) by enabling distributed, overlapping memory representations.
• To integrate a memory system with deep generative models by formulating it as a hierarchical conditional generative model with a data-dependent, adaptive prior.
• To enable efficient, principled memory updates via an exact Bayesian update rule that balances retention of old content and storage of new data.
• To demonstrate that the model improves generative modeling performance on Omniglot and CIFAR-10 while being significantly easier to train than DNCs.

Proposed method

• The model extends the VAE framework by replacing the standard prior with a memory-dependent prior derived from a generative, distributed memory store.
• It uses a hierarchical conditional generative model where the memory provides a rich, data-adaptive prior, and the latent code is formed by combining top-down memory signals with bottom-up perception from an encoder.
• The memory is implemented as a linear Gaussian model with learnable addresses and reparameterized latent variables, enabling differentiable, analytically tractable inference.
• A key innovation is the derivation of a Bayesian update rule for memory writing that optimally trades off preserving old content and incorporating new data, ensuring robust online compression.
• The model is trained end-to-end using a variational lower bound on the log-likelihood of exchangeable episodes, with memory states updated via exact Bayesian inference.
• The memory is updated using a closed-form solution based on Bayes' rule, avoiding gradient-based optimization of the memory update itself, which enhances training stability.

Experimental results

Research questions

• RQ1 Can a memory system be designed to adapt quickly to new data while maintaining robustness and scalability in training?
• RQ2 How can a memory model achieve high capacity and efficient compression without relying on raw pixel storage or slot-based architectures?
• RQ3 Can an analytically tractable Bayesian update rule be used to enable optimal, differentiable memory writing in a deep generative model?
• RQ4 Does a memory-augmented VAE with a data-dependent, adaptive prior improve generative modeling performance on few-shot and standard image datasets?
• RQ5 How does the Kanerva Machine compare in training stability, convergence speed, and generalization capacity to existing models like the DNC and Neural Statistician?

Key findings

• The Kanerva Machine achieved a test loss below 70 on CIFAR-10 across all tested hyperparameter configurations, significantly outperforming the DNC, which only reached a loss near 100 in 2 out of 6 runs with optimal settings.
• The model demonstrated robust training across a wide range of learning rates (3×10⁻⁵ to 3×10⁻⁴) and batch sizes (8 to 64), unlike the DNC, which was highly sensitive to hyperparameters and initialization.
• The Kanerva Machine maintained a clear advantage over the DNC in generalization to larger episodes, with lower variational lower bounds on reconstruction loss, especially when episodes contained fewer classes (higher redundancy).
• The model successfully leveraged statistical regularities in data by learning compressed representations through the encoder, addresses, and Bayesian updates, avoiding the need to store raw pixels.
• The use of an exact Bayesian update rule enabled principled, parameter-free memory writing that did not require backpropagation through the update process, enhancing training stability and efficiency.
• The hierarchical generative model with memory-dependent priors enabled effective sampling of unseen patterns, consistent with constructive memory phenomena observed in neuroscience.