[Paper Review] The Neural Coding Framework for Learning Generative Models

Alexander G. Ororbia, Daniel Kifer|arXiv (Cornell University)|Dec 7, 2020

Neural Networks and Applications4 citations

TL;DR

This paper proposes the Neural Generative Coding (NGC) framework, a biologically inspired alternative to backpropagation for training deep generative models. By modeling neurons as making hierarchical predictions and updating weights via local error signals—mirroring predictive processing in the brain—NGC achieves competitive performance on generative tasks without backprop, demonstrating strong results in image generation, representation learning, and pattern completion.

ABSTRACT

Neural generative models can be used to learn complex probability distributions from data, to sample from them, and to produce probability density estimates. We propose a computational framework for developing neural generative models inspired by the theory of predictive processing in the brain. According to predictive processing theory, the neurons in the brain form a hierarchy in which neurons in one level form expectations about sensory inputs from another level. These neurons update their local models based on differences between their expectations and the observed signals. In a similar way, artificial neurons in our generative models predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality. In this work, we show that the neural generative models learned within our framework perform well in practice across several benchmark datasets and metrics and either remain competitive with or significantly outperform other generative models with similar functionality (such as the variational auto-encoder).

Motivation & Objective

• To develop a biologically plausible learning framework for generative models that avoids reliance on backpropagation.
• To address the limitations of backpropagation in deep networks, such as non-local credit assignment and lack of lateral connectivity.
• To bridge the gap between neuroscience-inspired principles and practical machine learning by modeling neural prediction and error correction in a hierarchical, local manner.
• To demonstrate that a framework based on predictive processing can achieve competitive performance in generative modeling tasks.
• To explore whether local, hierarchical, and predictive learning mechanisms can yield general-purpose representations suitable for downstream tasks.

Proposed method

• Proposes a hierarchical neural network architecture where each layer predicts the activity of the layer below using generative synaptic weights.
• Employs local, error-driven synaptic updates based on prediction errors, inspired by predictive processing and free energy minimization.
• Uses a variational inference framework to minimize variational free energy, enabling end-to-end training without backpropagation.
• Introduces a message-passing mechanism where prediction errors are propagated backward and used to update synaptic weights locally.
• Applies Hebbian-like learning rules for synaptic plasticity, ensuring local and biologically plausible weight updates.
• Employs a two-stage training process: first, a recognition network infers latent states; second, the generative model synthesizes data using learned weights and prediction mechanisms.

Experimental results

Research questions

• RQ1 Can a biologically inspired learning framework based on predictive processing train deep generative models effectively without backpropagation?
• RQ2 How does local, hierarchical prediction and error-based learning compare to backpropagation in terms of generative performance and representation quality?
• RQ3 Can the proposed framework learn disentangled and meaningful representations that support downstream tasks like classification and pattern completion?
• RQ4 To what extent can the NGC framework recover or generalize known models such as those in [68] and [22]?
• RQ5 What are the implications of this framework for understanding credit assignment in the brain and for designing more biologically plausible artificial agents?

Key findings

• The NGC framework achieves competitive performance on benchmark datasets such as MNIST, CIFAR-10, and CelebA in image generation and density estimation tasks.
• NGC models demonstrate strong performance in downstream classification and pattern completion, indicating the learning of general-purpose, disentangled representations.
• Visualizations of learned synaptic weights reveal that higher layers form dynamic, context-sensitive maps that control the composition of low-level visual features in the output.
• The framework successfully recovers and generalizes known models from [68] and [22], showing theoretical consistency with existing variational inference frameworks.
• NGC models exhibit lateral competition and sparse activation patterns, suggesting the emergence of structured, biologically plausible representations.
• The framework supports end-to-end training without backpropagation, demonstrating that local, hierarchical, and predictive learning mechanisms can yield effective generative models.