[Paper Review] Fast and Functional Structured Data Generators Rooted in Out-of-Equilibrium Physics

Alessandra Carbone, Aurélien Decelle+2 more|arXiv (Cornell University)|Jul 13, 2023

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

This paper proposes a novel non-equilibrium training method for Restricted Boltzmann Machines (RBMs) that enables fast, high-quality conditional generation of structured data—such as genomic, protein, and RNA sequences—by leveraging out-of-equilibrium Markov chain dynamics. The method achieves state-of-the-art sample quality in just 10 MCMC steps, significantly outperforming equilibrium training in diversity, speed, and stability across diverse biological datasets.

ABSTRACT

In this study, we address the challenge of using energy-based models to produce high-quality, label-specific data in complex structured datasets, such as population genetics, RNA or protein sequences data. Traditional training methods encounter difficulties due to inefficient Markov chain Monte Carlo mixing, which affects the diversity of synthetic data and increases generation times. To address these issues, we use a novel training algorithm that exploits non-equilibrium effects. This approach, applied on the Restricted Boltzmann Machine, improves the model's ability to correctly classify samples and generate high-quality synthetic data in only a few sampling steps. The effectiveness of this method is demonstrated by its successful application to four different types of data: handwritten digits, mutations of human genomes classified by continental origin, functionally characterized sequences of an enzyme protein family, and homologous RNA sequences from specific taxonomies.

Motivation & Objective

• To address the limitations of traditional energy-based models in generating diverse, label-specific structured data due to poor MCMC mixing and slow convergence.
• To overcome the inefficiency and instability of equilibrium training in RBMs when applied to complex, multimodal biological datasets like genomics and protein sequences.
• To develop a training framework that enables rapid, high-fidelity conditional generation with minimal sampling steps, even for rare or subfamily-specific sequence classes.
• To demonstrate that non-equilibrium training of RBMs can function as a diffusion-like generator, improving sample diversity and generation speed while maintaining classification accuracy.
• To validate the method across diverse biological datasets, including handwritten digits, human genome mutations, enzyme protein families, and RNA sequences, with biologically meaningful structural validation.

Proposed method

• The method employs a two-gradient algorithm that trains the RBM using non-equilibrium Markov chain dynamics, avoiding full convergence to equilibrium.
• It uses a modified contrastive divergence approach where gradient estimation and sampling occur at non-equilibrium states, improving training stability and speed.
• The model is trained to generate label-conditioned samples by incorporating class labels into the energy function, enabling conditional generation from random initializations.
• Sampling is performed with only 10 MCMC steps per generation, significantly reducing inference time while maintaining high sample quality.
• The approach is applied to RBMs with a simple energy function, but the framework is extensible to more complex architectures like those with convolutional layers.
• The method is evaluated using multiple metrics: eigenvalue spectrum error, entropy error, and adversarial accuracy, with biological validation via ESMFold-predicted pLDDT scores.

Experimental results

Research questions

• RQ1 Can non-equilibrium training of RBMs generate high-quality, diverse, and label-specific synthetic data in complex structured datasets like genomics and proteomics?
• RQ2 Does training RBMs out of equilibrium improve generation speed and sample diversity compared to standard equilibrium methods?
• RQ3 Can the model generate biologically plausible protein sequences with correct functional and structural properties, even when trained on limited data per subfamily?
• RQ4 How does the performance of the F&F-10 model compare to equilibrium RBM training in terms of generation quality and sampling efficiency?
• RQ5 To what extent can the model generalize to rare or low-frequency sequence categories with only a few training examples?

Key findings

• The F&F-10 model generated high-quality synthetic samples in just 10 MCMC steps, matching the diversity and structure of real data across all tested datasets, including MNIST, HGD, GH30, and SAM.
• The model achieved high classification accuracy on all datasets, even for subfamilies with as few as 100 training examples, demonstrating robust generalization.
• Generated protein sequences showed pLDDT score distributions nearly identical to real test data (as measured by ESMFold), indicating high structural reliability and biological plausibility.
• The method reduced training time and improved stability compared to standard persistent contrastive divergence (PCD) training, which often fails to mix properly in multimodal distributions.
• The F&F-10 model outperformed equilibrium RBM training in both sample quality and generation speed, with minimal sampling steps required for high-fidelity output.
• The two-gradient non-equilibrium training strategy enabled effective conditional generation without requiring long MCMC chains, making it suitable for real-world biological data generation tasks.