[Paper Review] Physical Deep Learning with Biologically Plausible Training Method

M. Nakajima, Katsuma Inoue+4 more|arXiv (Cornell University)|Apr 1, 2022

Neural Networks and Reservoir Computing4 citations

TL;DR

This paper proposes a biologically plausible training method for physical deep neural networks using direct feedback alignment with random projections, enabling end-to-end training without backpropagation. The approach is demonstrated on an optoelectronic recurrent network, achieving competitive performance with accelerated computation on an FPGA-based benchtop system.

ABSTRACT

The ever-growing demand for further advances in artificial intelligence motivated research on unconventional computation based on analog physical devices. While such computation devices mimic brain-inspired analog information processing, learning procedures still relies on methods optimized for digital processing such as backpropagation. Here, we present physical deep learning by extending a biologically plausible training algorithm called direct feedback alignment. As the proposed method is based on random projection with arbitrary nonlinear activation, we can train a physical neural network without knowledge about the physical system. In addition, we can emulate and accelerate the computation for this training on a simple and scalable physical system. We demonstrate the proof-of-concept using a hierarchically connected optoelectronic recurrent neural network called deep reservoir computer. By constructing an FPGA-assisted optoelectronic benchtop, we confirmed the potential for accelerated computation with competitive performance on benchmarks. Our results provide practical solutions for the training and acceleration of neuromorphic computation.

Motivation & Objective

• To develop a training method for physical neural networks that is biologically plausible and avoids backpropagation.
• To enable training of physical systems without requiring detailed knowledge of their internal dynamics.
• To demonstrate scalable and efficient training using random projections and arbitrary nonlinearities.
• To accelerate physical deep learning by emulating the training process on a simple, scalable physical system.
• To validate the approach on a real-world optoelectronic recurrent neural network platform.

Proposed method

• The method extends direct feedback alignment (DFA) to physical neural networks, using random projections to approximate error signals.
• It enables end-to-end training without backpropagation, relying only on random weight matrices and nonlinear activation functions.
• The training process is designed to be independent of the physical system's internal structure, making it applicable to diverse analog devices.
• The framework allows emulation of the training process on a scalable physical system, such as an optoelectronic reservoir computer.
• An FPGA-assisted optoelectronic benchtop system is used to implement and validate the method in hardware.
• The approach supports hierarchical connectivity and leverages the physical dynamics of the system for computation.

Experimental results

Research questions

• RQ1 Can direct feedback alignment be adapted to train physical deep neural networks without backpropagation?
• RQ2 Can random projections enable effective error signal approximation in physical systems with unknown dynamics?
• RQ3 Can the training method be implemented and accelerated on a real optoelectronic hardware platform?
• RQ4 Does the proposed method achieve competitive performance on standard benchmarks using physical computation?
• RQ5 Can the training process be emulated efficiently on a scalable physical system?

Key findings

• The proposed method successfully trains a physical deep neural network using biologically plausible learning rules without requiring backpropagation.
• The method achieves competitive performance on benchmark tasks using an optoelectronic recurrent neural network on an FPGA-based benchtop system.
• The training process is scalable and does not require detailed knowledge of the physical system's internal dynamics.
• Random projections enable effective error signal approximation, supporting stable and efficient learning in physical systems.
• The system demonstrates accelerated computation, confirming the feasibility of physical deep learning with biologically plausible training.
• The approach enables emulation and hardware acceleration of the training process on a simple, scalable physical platform.