[Paper Review] On the Infinite Width and Depth Limits of Predictive Coding Networks

Francesco Innocenti, El Mehdi Achour+1 more|arXiv (Cornell University)|Feb 7, 2026

Single-cell and spatial transcriptomics0 citations

TL;DR

The paper proves that for linear predictive coding networks (PCNs), the set of width-stable and feature-learning parameterisations matches that of backpropagation (BP); in the limit of wide models (N >> L), PC gradients converge to BP gradients, and this extends to deep residual networks with appropriate depth scaling. Experiments on linear and nonlinear networks support these limits, showing PC converges to BP when activity equilibria are reached.

ABSTRACT

Predictive coding (PC) is a biologically plausible alternative to standard backpropagation (BP) that minimises an energy function with respect to network activities before updating weights. Recent work has improved the training stability of deep PC networks (PCNs) by leveraging some BP-inspired reparameterisations. However, the full scalability and theoretical basis of these approaches remains unclear. To address this, we study the infinite width and depth limits of PCNs. For linear residual networks, we show that the set of width- and depth-stable feature-learning parameterisations for PC is exactly the same as for BP. Moreover, under any of these parameterisations, the PC energy with equilibrated activities converges to the BP loss in a regime where the model width is much larger than the depth, resulting in PC computing the same gradients as BP. Experiments show that these results hold in practice for deep nonlinear networks, as long as an activity equilibrium seem to be reached. Overall, this work unifies various previous theoretical and empirical results and has potentially important implications for the scaling of PCNs.

Motivation & Objective

• Motivate understanding of credit assignment in neural circuits by connecting predictive coding to backpropagation.
• Derive principled width- and depth-aware parameterisations for PCNs that yield stable learning.
• Establish theoretical equivalence (in wide regimes) between PCN and BP gradients in linear and residual networks.
• Extend insights to nonlinear networks with empirical validation.
• Discuss implications for scaling PCNs and biological plausibility of learning rules.

Proposed method

• Define a general width-parameterisation for linear MLPs with scaling exponents a_b, b_b, c, d and width N.
• Use equilibrated PC energy F^*() to relate PC learning to a rescaled MSE loss via s().
• Prove Theorem 1: width-stable and feature-learning parameterisations for PC match those of BP.
• Show Corollary 3.2: as N G ty, F^*()  L() and PC gradients converge to BP gradients.
• Extend parameterisations to linear residual networks; prove Theorem 2: depth-aware parameterisations for PC match BP (α = 1/2).
• Provide empirical verification on linear and nonlinear networks; demonstrate convergence to BP for wide models and equilibrated activities (Figure 4).
• Discuss implications for stability, scaling, and biological plausibility of PCNs; outline limitations and future work.

Figure 1 : Under width- and depth-stable feature-learning parameterisations of linear residual networks, PC converges to BP when the model width is much larger than the depth, $N\gg L$ . We trained linear residual networks on CIFAR-10 with the mean-field parameterisation (as defined in Table 2 ) and

Experimental results

Research questions

• RQ1 Can principled width- and depth-aware parameterisations for PCNs be derived that yield stable learning?
• RQ2 Do PC gradients converge to BP gradients in the infinite width or depth limits under such parameterisations?
• RQ3 Do these results extend from linear to nonlinear PCNs with practical equilibrated activities?
• RQ4 What are the practical implications for scaling PCNs and for biological plausibility of learning rules?

Key findings

• Theorem 1: For linear MLPs with equilibrated activities, the width-stable and feature-learning parameterisations for PC coincide with those for BP.
• Corollary 3.2: Under these parameterisations, PC gradients converge to BP gradients as width N ty (s()  1).
• Theorem 2: For linear residual networks, the width- and depth-stable feature-learning parameterisations for PC match those for BP (α = 1/2).
• Corollary 4.2: In the regime N ty and L ty with N e L (N ty, L finite or relatively smaller), PC gradients converge to BP gradients.
• Figure 4 and accompanying results show that nonlinear networks reach BP-like gradients when width is much larger than depth and activities equilibrate.
• The work provides a unified view linking infinite-width/depth analyses to practical PCN training and stability.

Figure 2 : Under width-stable and feature-learning parameterisations of linear MLPs, PC converges to BP at large width. We trained deep linear MLPs ( $L=5$ ) of varying widths $N$ with full-batch GD on a toy task with binary labels. All models used the mean-field parameterisation as defined in Table