[论文解读] A Theoretical Framework for Target Propagation

Alexander Meulemans, Francesco S. Carzaniga+3 more|arXiv (Cornell University)|Jun 25, 2020

Wireless Signal Modulation Classification参考文献 61被引用 32

一句话总结

本论文构建了理论框架，表明 Target Propagation (TP) 在可逆网络中充当高斯-牛顿–梯度混合，并引入差分重构损失（DRL）以在非可逆网络中实现 GN 类目标，外加直接反馈变体，实验结果显示显著提升。

ABSTRACT

The success of deep learning, a brain-inspired form of AI, has sparked interest in understanding how the brain could similarly learn across multiple layers of neurons. However, the majority of biologically-plausible learning algorithms have not yet reached the performance of backpropagation (BP), nor are they built on strong theoretical foundations. Here, we analyze target propagation (TP), a popular but not yet fully understood alternative to BP, from the standpoint of mathematical optimization. Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. Furthermore, our analysis reveals a fundamental limitation of difference target propagation (DTP), a well-known variant of TP, in the realistic scenario of non-invertible neural networks. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training, while simultaneously introducing architectural flexibility by allowing for direct feedback connections from the output to each hidden layer. Our theory is corroborated by experimental results that show significant improvements in performance and in the alignment of forward weight updates with loss gradients, compared to DTP.

研究动机与目标

• Motivate and analyze TP as an optimization framework distinct from backpropagation (BP).
• Characterize TP in invertible networks as a Gauss-Newton–gradient descent hybrid.
• Identify limitations of Difference Target Propagation (DTP) in non-invertible networks.
• Propose a novel Difference Reconstruction Loss (DRL) to propagate Gauss-Newton targets in non-invertible nets.
• Introduce direct feedback connections and evaluate DPTP variants experimentally.
• Show improved alignment of forward weight updates with loss gradients and GN targets.

提出的方法

• Formalize invertible network conditions and derive TP target updates as GN-like transformations (Theorem 2).
• Show limitations of DTP in non-invertible networks due to reconstruction error (Lemma 3).
• Introduce Difference Reconstruction Loss (DRL) to train feedback mappings so they propagate GN targets (Eq. 10, Theorem 4).
• Propose Direct Difference Target Propagation (DDTP) variants with direct output-to-hidden feedback (DDTP-linear, DDTP-RHL).
• Provide theoretical interpretation of Gauss-Newton Target (GNT) updates and minimum-norm properties (Theorems 5-6).
• Empirically compare TP, DTP, DDTP variants on MNIST, Frozen-MNIST, Fashion-MNIST, CIFAR-10 using FC and small CNN models.

实验结果

研究问题

• RQ1 Can TP be rigorously related to Gauss-Newton optimization in invertible networks?
• RQ2 Why does DTP underperform in non-invertible networks, and can this be remedied?
• RQ3 Does DRL enable propagation of GN-like targets in non-invertible networks?
• RQ4 Do direct feedback connections (DDTP variants) improve learning signals and performance?
• RQ5 How do GN-target-based updates compare to BP in terms of convergence and norm-optimality?

主要发现

• TP targets in invertible networks implement a Gauss-Newton–gradient descent hybrid update.
• DTP suffers from ineffective updates in non-invertible networks due to reconstruction errors (interference with target propagation).
• A novel Difference Reconstruction Loss (DRL) trains feedback mappings to propagate Gauss-Newton targets, mitigating DTP limitations.
• Direct feedback connections (DDTP variants) improve learning signals and matching to loss gradients and GN targets.
• DDTP-linear and related variants outperform original DTP and controls across MNIST, Fashion-MNIST, and CIFAR-10 in test errors; DDTP-linear often strongest overall.
• GNT updates align with GN directions and can yield minimum-norm updates in linear networks (and approximate behavior in nonlinear networks).