[Paper Review] Associative Memory in Iterated Overparameterized Sigmoid Autoencoders

Yibo Jiang, Cengiz Pehlevan|arXiv (Cornell University)|Jun 30, 2020

Model Reduction and Neural Networks4 citations

TL;DR

This paper investigates associative memory in overparameterized sigmoid autoencoders using the Neural Tangent Kernel (NTK) framework. It shows that in the infinite-width limit, iterative training leads to stable fixed points (attractors) when the input-output Jacobian's largest eigenvalue norm drops below one—achievable with increasing input norm—enabling robust associative memory for both single and multiple training examples.

ABSTRACT

Recent work showed that overparameterized autoencoders can be trained to implement associative memory via iterative maps, when the trained input-output Jacobian of the network has all of its eigenvalue norms strictly below one. Here, we theoretically analyze this phenomenon for sigmoid networks by leveraging recent developments in deep learning theory, especially the correspondence between training neural networks in the infinite-width limit and performing kernel regression with the Neural Tangent Kernel (NTK). We find that overparameterized sigmoid autoencoders can have attractors in the NTK limit for both training with a single example and multiple examples under certain conditions. In particular, for multiple training examples, we find that the norm of the largest Jacobian eigenvalue drops below one with increasing input norm, leading to associative memory.

Motivation & Objective

• To understand how overparameterized sigmoid autoencoders can implement associative memory through iterative dynamics.
• To analyze the conditions under which stable fixed points (attractors) emerge in the infinite-width limit.
• To investigate the role of the input-output Jacobian's eigenvalue spectrum in determining memory stability.
• To determine whether associative memory functionality holds for both single and multiple training examples.
• To leverage the Neural Tangent Kernel (NTK) framework to derive theoretical guarantees for convergence and stability.

Proposed method

• Utilizes the Neural Tangent Kernel (NTK) framework to analyze the infinite-width limit of overparameterized sigmoid autoencoders.
• Derives the input-output Jacobian of the autoencoder in the NTK regime to study its spectral properties.
• Analyzes the norm of the largest eigenvalue of the Jacobian as a function of input norm to determine convergence conditions.
• Applies iterative map dynamics to model the network's behavior during inference, treating it as a fixed-point iteration.
• Considers both single-example and multi-example training scenarios to assess generalization of associative memory behavior.
• Establishes theoretical conditions under which the Jacobian's spectral norm drops below one, ensuring convergence to stable attractors.

Experimental results

Research questions

• RQ1 Under what conditions does the input-output Jacobian of an overparameterized sigmoid autoencoder have all eigenvalues with norm strictly less than one?
• RQ2 Can associative memory emerge in the NTK limit for both single and multiple training examples?
• RQ3 How does increasing input norm affect the spectral norm of the Jacobian and the stability of fixed points?
• RQ4 What role does the NTK play in enabling iterative convergence to attractors in overparameterized networks?
• RQ5 Is there a theoretical guarantee for stable associative memory in sigmoid autoencoders under overparameterization?

Key findings

• In the NTK limit, overparameterized sigmoid autoencoders can achieve stable fixed points (attractors) when the input-output Jacobian's largest eigenvalue norm is strictly less than one.
• For multiple training examples, the norm of the largest Jacobian eigenvalue decreases below one as the input norm increases, enabling associative memory functionality.
• Theoretical analysis confirms that iterative maps in the NTK regime converge to stable attractors under the derived spectral conditions.
• Associative memory is supported not only for single-example training but also for multi-example scenarios under the same spectral constraints.
• The NTK framework provides a rigorous theoretical foundation for understanding the dynamics of overparameterized autoencoders as associative memory systems.
• The results demonstrate that overparameterization combined with sigmoid activation enables robust, stable memory retrieval via iterative refinement.