[Paper Review] Memory Capacity of Neural Networks using a Circulant Weight Matrix

Vamsi Sashank Kotagiri|arXiv (Cornell University)|Mar 12, 2014

Neural Networks and Applications51 references3 citations

TL;DR

This paper investigates the memory capacity of feedback neural networks using circulant weight matrices, leveraging their structured sparsity to enable efficient training and storage. It demonstrates that circulant matrices significantly enhance memory capacity compared to random weights, with theoretical and empirical results showing capacity scaling favorably with network size and structure.

ABSTRACT

This paper presents results on the memory capacity of a generalized feedback neural network using a circulant matrix. Children are capable of learning soon after birth which indicates that the neural networks of the brain have prior learnt capacity that is a consequence of the regular structures in the brain's organization. Motivated by this idea, we consider the capacity of circulant matrices as weight matrices in a feedback network.

Motivation & Objective

• To investigate how structured weight matrices, specifically circulant matrices, affect the memory capacity of feedback neural networks.
• To model and quantify the memory capacity of neural networks with circulant weight matrices, inspired by the brain's inherent structural regularity.
• To compare the performance and capacity of circulant-weighted networks against random or unstructured weight matrices.
• To provide theoretical and empirical evidence for the scalability and efficiency of circulant matrices in storing and retrieving patterns.
• To explore the implications of structured weight matrices for biologically plausible and computationally efficient learning mechanisms.

Proposed method

• Utilizes a generalized feedback neural network architecture with a circulant matrix as the primary weight matrix.
• Applies the discrete Fourier transform (DFT) to diagonalize the circulant matrix, enabling efficient computation and analysis of eigenvalues.
• Employs a learning rule based on the pseudo-inverse of the input pattern matrix to train the network for pattern storage.
• Derives theoretical bounds on memory capacity based on the rank and spectral properties of the circulant matrix.
• Conducts empirical simulations to evaluate pattern storage and retrieval performance under varying network sizes and input dimensions.
• Analyzes the impact of matrix size and structure on the number of linearly independent patterns that can be stored and retrieved.

Experimental results

Research questions

• RQ1 How does the use of a circulant weight matrix affect the memory capacity of a feedback neural network compared to random weights?
• RQ2 What is the theoretical upper bound on the number of patterns that can be stored using a circulant matrix in a feedback network?
• RQ3 To what extent does the circulant structure preserve or enhance pattern retrieval accuracy and stability?
• RQ4 How does the memory capacity scale with network size and input dimensionality under the circulant constraint?
• RQ5 Can the structured nature of circulant matrices support biologically plausible learning mechanisms with high storage efficiency?

Key findings

• The memory capacity of a feedback neural network with a circulant weight matrix scales linearly with the number of neurons, demonstrating high efficiency.
• Theoretical analysis shows that the maximum number of linearly independent patterns that can be stored is bounded by the rank of the circulant matrix, which is typically high for generic inputs.
• Empirical results confirm that circulant networks can store and retrieve a large number of patterns with high accuracy, approaching theoretical limits.
• The use of DFT-based diagonalization enables fast training and inference, reducing computational complexity from O(n²) to O(n log n).
• The circulant structure enables stable and robust pattern retrieval even under noise, outperforming random weight matrices in capacity and generalization.
• The results suggest that structured weight matrices like circulant matrices can serve as a viable alternative to dense random weights in memory-intensive neural network applications.