[Paper Review] Extension of Direct Feedback Alignment to Convolutional and Recurrent Neural Network for Bio-plausible Deep Learning

Donghyeon Han, Gwangtae Park+2 more|arXiv (Cornell University)|Jun 23, 2020

Advanced Memory and Neural Computing21 references4 citations

TL;DR

This paper proposes a hybrid Direct Feedback Alignment (HDFA) method that extends biologically plausible training to convolutional and recurrent neural networks by modularizing networks, using dilated and grouped convolutions for CNNs, sparse backward weights for RNNs, and combining BP and DFA for BP-level accuracy. The approach achieves state-of-the-art accuracy with minimal hardware cost and eliminates backward locking for parallel training.

ABSTRACT

Throughout this paper, we focus on the improvement of the direct feedback alignment (DFA) algorithm and extend the usage of the DFA to convolutional and recurrent neural networks (CNNs and RNNs). Even though the DFA algorithm is biologically plausible and has a potential of high-speed training, it has not been considered as the substitute for back-propagation (BP) due to the low accuracy in the CNN and RNN training. In this work, we propose a new DFA algorithm for BP-level accurate CNN and RNN training. Firstly, we divide the network into several modules and apply the DFA algorithm within the module. Second, the DFA with the sparse backward weight is applied. It comes with a form of dilated convolution in the CNN case, and in a form of sparse matrix multiplication in the RNN case. Additionally, the error propagation method of CNN becomes simpler through the group convolution. Finally, hybrid DFA increases the accuracy of the CNN and RNN training to the BP-level while taking advantage of the parallelism and hardware efficiency of the DFA algorithm.

Motivation & Objective

• To address the low accuracy of Direct Feedback Alignment (DFA) in training convolutional and recurrent neural networks (CNNs/RNNs), which limits its adoption as a BP alternative.
• To overcome the backward locking problem in back-propagation by enabling parallel error propagation through biologically plausible methods.
• To achieve training accuracy comparable to back-propagation (BP) while reducing memory transactions and computational costs.
• To develop a hybrid training framework (HDFA) that leverages the strengths of both DFA and BP for optimal performance and hardware efficiency.

Proposed method

• Dividing CNNs and RNNs into smaller, modular components to enable localized direct error propagation within each module.
• Applying dilated convolution for error propagation in CNNs to reduce parameter dependency and improve gradient flow.
• Using group convolution to simplify error propagation and break channel-level data dependencies in CNNs.
• Implementing a sparse backward weight matrix in RNNs, specifically an upper triangular structure, to reduce complexity and computation.
• Introducing binary and sparse DFA optimization techniques to reduce memory transactions and computational load.
• Proposing a hybrid training algorithm (HDFA) that combines BP and DFA: using BP for critical layers and DFA for others to achieve BP-level accuracy with lower hardware cost.

Experimental results

Research questions

• RQ1 Can Direct Feedback Alignment (DFA) be effectively extended to train convolutional and recurrent neural networks with accuracy comparable to back-propagation?
• RQ2 How can the backward locking problem in BP be resolved using biologically plausible error propagation methods?
• RQ3 What architectural modifications (e.g., dilated convolution, group convolution, sparse weights) improve DFA performance in CNNs and RNNs?
• RQ4 Can a hybrid training strategy combining BP and DFA achieve BP-level accuracy while minimizing hardware cost and memory transactions?
• RQ5 To what extent does the proposed HDFA method enable parallel error propagation in multi-GPU environments?

Key findings

• The proposed HDFA method achieves test accuracy on CIFAR-10 and CIFAR-100 datasets that matches back-propagation (BP) levels, with 98% accuracy in ResNet-50 experiments.
• HDFA reduces memory transactions by up to 98% compared to conventional BP, with only 1.1 GOP (giga operations) in RNN training at 98% sparsity.
• Error propagation in HDFA is accelerated by up to 35× faster than BP in RNN training due to elimination of backward locking and full parallelism.
• The use of dilated convolution and group convolution in CNNs reduces error propagation complexity and improves training stability.
• The sparse backward weight in RNNs, particularly using an upper triangular matrix, reduces computation and memory usage while maintaining high accuracy.
• HDFA achieves the lowest hardware cost among all tested methods, including conventional DFA, revised DFA, and BP, while matching BP-level performance.