[Paper Review] A Theoretical Framework for Target Propagation

Alexander Meulemans, Francesco S. Carzaniga+3 more|arXiv (Cornell University)|Jun 25, 2020

Wireless Signal Modulation Classification61 references32 citations

TL;DR

The paper develops a theoretical framework showing Target Propagation (TP) acts as a Gauss-Newton–gradient hybrid in invertible networks, and introduces a difference reconstruction loss (DRL) to enable GN-like targets in non-invertible networks, plus direct feedback variants with strong experimental gains.

ABSTRACT

The success of deep learning, a brain-inspired form of AI, has sparked interest in understanding how the brain could similarly learn across multiple layers of neurons. However, the majority of biologically-plausible learning algorithms have not yet reached the performance of backpropagation (BP), nor are they built on strong theoretical foundations. Here, we analyze target propagation (TP), a popular but not yet fully understood alternative to BP, from the standpoint of mathematical optimization. Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. Furthermore, our analysis reveals a fundamental limitation of difference target propagation (DTP), a well-known variant of TP, in the realistic scenario of non-invertible neural networks. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training, while simultaneously introducing architectural flexibility by allowing for direct feedback connections from the output to each hidden layer. Our theory is corroborated by experimental results that show significant improvements in performance and in the alignment of forward weight updates with loss gradients, compared to DTP.

Motivation & Objective

• Motivate and analyze TP as an optimization framework distinct from backpropagation (BP).
• Characterize TP in invertible networks as a Gauss-Newton–gradient descent hybrid.
• Identify limitations of Difference Target Propagation (DTP) in non-invertible networks.
• Propose a novel Difference Reconstruction Loss (DRL) to propagate Gauss-Newton targets in non-invertible nets.
• Introduce direct feedback connections and evaluate DPTP variants experimentally.
• Show improved alignment of forward weight updates with loss gradients and GN targets.

Proposed method

• Formalize invertible network conditions and derive TP target updates as GN-like transformations (Theorem 2).
• Show limitations of DTP in non-invertible networks due to reconstruction error (Lemma 3).
• Introduce Difference Reconstruction Loss (DRL) to train feedback mappings so they propagate GN targets (Eq. 10, Theorem 4).
• Propose Direct Difference Target Propagation (DDTP) variants with direct output-to-hidden feedback (DDTP-linear, DDTP-RHL).
• Provide theoretical interpretation of Gauss-Newton Target (GNT) updates and minimum-norm properties (Theorems 5-6).
• Empirically compare TP, DTP, DDTP variants on MNIST, Frozen-MNIST, Fashion-MNIST, CIFAR-10 using FC and small CNN models.

Experimental results

Research questions

• RQ1 Can TP be rigorously related to Gauss-Newton optimization in invertible networks?
• RQ2 Why does DTP underperform in non-invertible networks, and can this be remedied?
• RQ3 Does DRL enable propagation of GN-like targets in non-invertible networks?
• RQ4 Do direct feedback connections (DDTP variants) improve learning signals and performance?
• RQ5 How do GN-target-based updates compare to BP in terms of convergence and norm-optimality?

Key findings

• TP targets in invertible networks implement a Gauss-Newton–gradient descent hybrid update.
• DTP suffers from ineffective updates in non-invertible networks due to reconstruction errors (interference with target propagation).
• A novel Difference Reconstruction Loss (DRL) trains feedback mappings to propagate Gauss-Newton targets, mitigating DTP limitations.
• Direct feedback connections (DDTP variants) improve learning signals and matching to loss gradients and GN targets.
• DDTP-linear and related variants outperform original DTP and controls across MNIST, Fashion-MNIST, and CIFAR-10 in test errors; DDTP-linear often strongest overall.
• GNT updates align with GN directions and can yield minimum-norm updates in linear networks (and approximate behavior in nonlinear networks).