[Paper Review] Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks

Tommaso Salvatori, Yuhang Song+3 more|arXiv (Cornell University)|Mar 5, 2021

Neural dynamics and brain functionNeuroscience32 references17 citations

TL;DR

This paper introduces Zero-Divergence Inference Learning (Z-IL), a biologically plausible algorithm that exactly replicates backpropagation (BP) in convolutional and recurrent neural networks. By reformulating inference learning to eliminate convergence requirements and reduce computational cost, Z-IL achieves BP-equivalent performance with only a minimal overhead compared to standard BP, making it the first such method to do so on complex architectures like CNNs and RNNs.

ABSTRACT

Predictive coding networks (PCNs) are an influential model for information processing in the brain. They have appealing theoretical interpretations and offer a single mechanism that accounts for diverse perceptual phenomena of the brain. On the other hand, backpropagation (BP) is commonly regarded to be the most successful learning method in modern machine learning. Thus, it is exciting that recent work formulates inference learning (IL) that trains PCNs to approximate BP. However, there are several remaining critical issues: (i) IL is an approximation to BP with unrealistic/non-trivial requirements, (ii) IL approximates BP in single-step weight updates; whether it leads to the same point as BP after the weight updates are conducted for more steps is unknown, and (iii) IL is computationally significantly more costly than BP. To solve these issues, a variant of IL that is strictly equivalent to BP in fully connected networks has been proposed. In this work, we build on this result by showing that it also holds for more complex architectures, namely, convolutional neural networks and (many-to-one) recurrent neural networks. To our knowledge, we are the first to show that a biologically plausible algorithm is able to exactly replicate the accuracy of BP on such complex architectures, bridging the existing gap between IL and BP, and setting an unprecedented performance for PCNs, which can now be considered as efficient alternatives to BP.

Motivation & Objective

• To address the limitations of inference learning (IL), which approximates backpropagation (BP) with non-trivial convergence requirements and high computational cost.
• To close the gap between biologically plausible learning (via predictive coding networks) and the success of BP in complex deep learning models.
• To demonstrate that a variant of IL—Z-IL—can exactly replicate BP dynamics in convolutional and recurrent neural networks.
• To show that Z-IL is computationally efficient, reducing inference steps from T ≈ 100–200 (as in IL) to l_max, the minimal depth for signal propagation.
• To establish Z-IL as a viable, high-performance alternative to BP in both machine learning and neuroscience contexts.

Proposed method

• Z-IL reformulates inference learning by deriving exact weight update equations that match BP for convolutional and many-to-one recurrent neural networks.
• The method uses a modified inference process that computes prediction errors across layers without requiring convergence to equilibrium, eliminating the need for iterative inference until stability.
• It introduces a novel feedback mechanism that ensures the error signal propagates exactly as in BP, even in non-linear, deep architectures.
• The algorithm computes weight updates based on the minimal number of inference steps (l_max) required to transmit information from input to output, rather than requiring T steps as in IL.
• Theoretical derivations are grounded in free-energy minimization and variational inference, preserving biological plausibility while achieving mathematical equivalence to BP.
• Empirical validation is performed across multiple architectures (CNNs, RNNs) and datasets (FashionMNIST, ImageNet, Atari games), comparing running times and accuracy.

Experimental results

Research questions

• RQ1 Can a biologically plausible learning algorithm exactly replicate the weight updates of backpropagation in complex deep learning architectures like convolutional and recurrent networks?
• RQ2 Does Z-IL eliminate the need for convergence to equilibrium during inference, which is a non-trivial and unrealistic requirement in previous IL methods?
• RQ3 Can Z-IL achieve BP-equivalent performance with significantly reduced computational cost compared to standard inference learning?
• RQ4 Is the equivalence between Z-IL and BP preserved over multiple weight update steps, not just in a single update?
• RQ5 Does Z-IL maintain biological plausibility while matching the accuracy and efficiency of BP in large-scale tasks such as ImageNet classification?

Key findings

• Z-IL exactly replicates the weight updates of backpropagation in both convolutional and recurrent neural networks, proving mathematical equivalence.
• The method reduces the number of required inference steps from T ≈ 100–200 (as in IL) to l_max, the minimal depth for signal propagation, significantly lowering computational cost.
• On average, Z-IL runs in 3.81ms per weight update for ANNs, 5.67ms for RNNs, and 8.86ms for CNNs—only a minor overhead compared to BP (3.72ms, 5.64ms, 8.72ms respectively).
• In contrast, IL requires ~594ms (ANN), ~420ms (RNN), and ~661ms (CNN) per update, making it over 100x slower than BP.
• Z-IL achieves BP-level accuracy on ImageNet and Atari games, demonstrating that it is a practical and efficient alternative to BP in complex tasks.
• The results show that Z-IL is not only theoretically equivalent to BP but also computationally feasible, with orders-of-magnitude speedup over IL.