[论文解读] The Forward-Forward Algorithm: Some Preliminary Investigations

Geoffrey E. Hinton|arXiv (Cornell University)|Dec 27, 2022

Neural Networks and Applications被引用 117

一句话总结

Geoffrey Hinton 介绍 Forward-Forward (FF) 算法，一种使用正样本（真实数据）和负样本来训练多层网络的两前向传递学习过程，在小规模设置中在 MNIST 和 CIFAR-10 上显示出有竞争力的结果，并探索相较于反向传播在脑启发或低功耗硬件中的潜在优势。

ABSTRACT

The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the squared activities. If the positive and negative passes could be separated in time, the negative passes could be done offline, which would make the learning much simpler in the positive pass and allow video to be pipelined through the network without ever storing activities or stopping to propagate derivatives.

研究动机与目标

• 鉴于大脑的可行性以及实时处理约束，激发对一种可替代反向传播的学习过程的需求。
• 介绍 Forward-Forward (FF) 算法及其具有相反目标的两次前向传递。
• 在小规模问题（MNIST、CIFAR-10）上演示 FF，并分析其相对于反向传播的性能。
• 探索分层良好性度量和层归一化如何实现无需反向传播的多层学习。
• 讨论 FF 在类皮质学习和低功耗硬件方面的潜在优势。

提出的方法

• 定义一个逐层的良好性函数（例如活动的平方和），并使用两次前向传递：正数据（高良好性）和负数据（低良好性）。
• 对每一层以局部目标进行贪婪训练，使正数据的良好性增加、负数据的良好性减少。
• 对隐藏层的活动进行归一化，只传递它们的方向（长度保持作为良好性的量度，而不是绝对尺度）。
• 尝试平方活动良好性及其相反数，并应用层归一化以防止跨层信息泄露。
• 在 MNIST 上用多种体系结构评估 FF，包括每层有 2000 个 ReLU 的四层网络，并与反向传播基线进行比较。
• 将 FF 扩展到循环/时间处理以处理类似视频的输入，并讨论自顶向下与自底向上的一致性作为教学信号。

实验结果

研究问题

• RQ1 Can the Forward-Forward algorithm learn meaningful multi-layer representations without backpropagation?
• RQ2 How does FF perform on standard benchmarks (MNIST, CIFAR-10) compared to backpropagation under small-scale settings?
• RQ3 What roles do layer normalization and the choice of goodness function play in preventing information leakage between layers?
• RQ4 Can FF be extended to recurrent settings to model top-down influences and perceptual prediction?
• RQ5 What are the limitations and potential advantages of FF for cortex-like learning and low-power hardware?

主要发现

• FF can learn multi-layer representations using positive and negative data with layer-wise objectives.
• On MNIST, FF achieves competitive test errors in several setups, including supervised and unsupervised variants, with results approaching those of backpropagation in some configurations.
• In CIFAR-10 experiments with local receptive fields, FF is somewhat worse than backpropagation but still shows comparable performance when using ample hidden units and appropriate connectivity.
• Extending FF to recursive processing and using top-down context as a teaching signal yields reasonable performance on MNIST with recurrent architectures.
• FF demonstrates potential advantages for cortex-inspired learning and low-power analog hardware by avoiding backpropagation, while remaining slower and generally less accurate than backpropagation on the tested small-scale tasks.