[Paper Review] Hopfield model with planted patterns: a teacher-student self-supervised learning model

Francesco Alemanno, Luca Camanzi+2 more|arXiv (Cornell University)|Apr 26, 2023

Neural Networks and Applications4 citations

TL;DR

This paper introduces a generalized Hopfield model with planted correlated patterns to model teacher-student self-supervised learning, where spin variables represent neural network weights and patterns correspond to training examples. The key contribution is a phase diagram showing that memorization dominates with small, clean datasets, while generalization emerges above a critical threshold of noisy examples, revealing a transition from memory to learning driven by dataset noise and weight regularization (inference temperature).

ABSTRACT

While Hopfield networks are known as paradigmatic models for memory storage and retrieval, modern artificial intelligence systems mainly stand on the machine learning paradigm. We show that it is possible to formulate a teacher-student self-supervised learning problem with Boltzmann machines in terms of a suitable generalization of the Hopfield model with structured patterns, where the spin variables are the machine weights and patterns correspond to the training set's examples. We analyze the learning performance by studying the phase diagram in terms of the training set size, the dataset noise and the inference temperature (i.e. the weight regularization). With a small but informative dataset the machine can learn by memorization. With a noisy dataset, an extensive number of examples above a critical threshold is needed. In this regime the memory storage limits of the system becomes an opportunity for the occurrence of a learning regime in which the system can generalize.

Motivation & Objective

• To bridge associative memory in Hopfield networks with modern self-supervised machine learning by modeling weight dynamics as spin configurations.
• To investigate how dataset noise and weight regularization (inference temperature) influence the transition from memorization to generalization in learning.
• To derive a phase diagram that characterizes the learning regime in terms of training set size, noise level, and regularization strength.
• To establish a statistical mechanics framework for understanding generalization in Boltzmann machines via a teacher-student self-supervised setup.
• To analyze the critical thresholds at which generalization becomes possible despite noisy or weakly informative training data.

Proposed method

• Formulates a Hopfield model with N binary spin variables representing neural network weights and M quenched binary patterns representing training examples.
• Introduces a planted configuration (teacher signal) via an external field term λ∑iξ̂iξi in the Hamiltonian, modeling prior knowledge.
• Uses a Boltzmann-Gibbs distribution over spin configurations to represent the posterior distribution of network weights given the training data.
• Applies replica symmetric and cavity methods to compute the free energy and order parameters (e.g., overlap with planted pattern) in the thermodynamic limit.
• Derives self-consistent equations for the overlap and magnetization to analyze retrieval and generalization phases.
• Analyzes the system's behavior in both Bayes-optimal and non-optimal regimes to identify phase transitions between memorization and generalization.

Experimental results

Research questions

• RQ1 What is the critical threshold of training examples above which generalization becomes possible in a noisy dataset?
• RQ2 How does the interplay between dataset noise and weight regularization (inference temperature) affect the transition from memorization to generalization?
• RQ3 Can a Hopfield network with structured, correlated patterns exhibit a regime where it learns underlying data structure rather than merely memorizing examples?
• RQ4 What is the role of the planted configuration (teacher signal) in enabling generalization in the presence of noise?
• RQ5 How does the system's phase diagram—defined by training set size, noise, and regularization—determine the learning regime?

Key findings

• When the dataset is small and noise is low, the system operates in a memorization regime where it retrieves the planted pattern accurately.
• For noisy datasets, generalization becomes possible only when the number of training examples exceeds a critical threshold, which scales with the noise level.
• Above this threshold, the system transitions into a generalization regime where it can extract the underlying signal from weakly informative, noisy examples.
• The critical threshold for generalization is determined by the balance between dataset noise and weight regularization (inference temperature), with higher noise requiring more examples.
• The phase diagram reveals that memory storage capacity and generalization are not mutually exclusive; instead, high-capacity memory systems can support generalization when trained on sufficiently many noisy examples.
• Theoretical analysis confirms that the only stable solution for pattern overlap differences is zero, implying symmetry in retrieval when the system is in a generalization phase.