[Paper Review] Training Neural Networks Using Features Replay.

Zhouyuan Huo, Bin Gu+1 more|arXiv (Cornell University)|Jan 1, 2018

Machine Learning and ELMComputer Science38 citations

TL;DR

This paper proposes Features Replay, a novel parallel-objective formulation that decouples backpropagation to enable parallel training of deep neural networks. By ensuring convergence to critical points and reducing memory usage, the method achieves faster convergence, lower memory consumption, and improved generalization error compared to existing approaches.

ABSTRACT

Training a neural network using backpropagation algorithm requires passing error gradients sequentially through the network. The backward locking prevents us from updating network layers in parallel and fully leveraging the computing resources. Recently, there are several works trying to decouple and parallelize the backpropagation algorithm. However, all of them suffer from severe accuracy loss or memory explosion when the neural network is deep. To address these challenging issues, we propose a novel parallel-objective formulation for the objective function of the neural network. After that, we introduce features replay algorithm and prove that it is guaranteed to converge to critical points for the non-convex problem under certain conditions. Finally, we apply our method to training deep convolutional neural networks, and the experimental results show that the proposed method achieves {faster} convergence, {lower} memory consumption, and {better} generalization error than compared methods.

Motivation & Objective

• To address the sequential dependency in backpropagation that limits parallelization and efficient resource utilization in deep neural networks.
• To overcome the severe accuracy loss and memory explosion commonly encountered in existing parallel backpropagation methods when training deep networks.
• To develop a theoretically grounded, parallelizable training objective that guarantees convergence to critical points for non-convex problems.
• To reduce memory consumption during training while maintaining or improving model generalization performance.

Proposed method

• Proposes a new parallel-objective formulation of the neural network's objective function to decouple gradient computation across layers.
• Introduces the Features Replay algorithm, which replays intermediate feature representations to enable independent and parallel updates of network layers.
• Proves convergence to critical points under certain conditions, ensuring theoretical reliability for non-convex optimization problems.
• Leverages feature replay to avoid the need for full backward pass storage, significantly reducing memory overhead.
• Designs the training process to allow layer-wise updates independent of sequential backpropagation, enabling efficient GPU utilization.
• Maintains model accuracy by preserving gradient flow through replayed features, avoiding the degradation seen in prior parallel methods.

Experimental results

Research questions

• RQ1 Can a parallel-objective formulation for neural network training achieve convergence to critical points in non-convex settings?
• RQ2 Can the proposed method reduce memory consumption during backpropagation without sacrificing model accuracy?
• RQ3 Does the Features Replay algorithm enable faster convergence compared to standard backpropagation and existing parallel methods?
• RQ4 Can the method generalize better than baseline approaches on deep convolutional neural networks?
• RQ5 Is the method scalable and stable when applied to very deep networks?

Key findings

• The Features Replay algorithm achieves faster convergence than standard backpropagation and existing parallel methods.
• Memory consumption is significantly reduced due to the elimination of full backward pass storage requirements.
• The method demonstrates better generalization error on deep convolutional neural networks compared to baseline approaches.
• Theoretical analysis confirms convergence to critical points under specified conditions, validating the method's stability.
• Experimental results show consistent performance gains across multiple deep learning benchmarks, with measurable improvements in training speed and memory efficiency.