[Paper Review] Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging

Daniel Povey, Xiaohui Zhang+1 more|arXiv (Cornell University)|Jan 1, 2014

Neural Networks and Applications28 references105 citations

TL;DR

This paper presents a hardware-agnostic framework for training deep neural networks (DNNs) across multiple GPUs or multicore machines using periodic parameter averaging and an efficient approximation of Natural Gradient (NG-SGD). The method enables scalable, low-communication distributed training that significantly improves convergence on single machines and maintains performance in multi-node settings.

ABSTRACT

We describe the neural-network training framework used in the Kaldi speech recognition toolkit, which is geared towards training DNNs with large amounts of training data using multiple GPU-equipped or multicore machines. In order to be as hardwareagnostic as possible, we needed a way to use multiple machines without generating excessive network traffic. Our method is to average the neural network parameters periodically (typically every minute or two), and redistribute the averaged parameters to the machines for further training. Each machine sees different data. By itself, this method does not work very well. However, we have another method, an approximate and efficient implementation of Natural Gradient for Stochastic Gradient Descent (NG-SGD), which seems to allow our periodic-averaging method to work well, as well as substantially improving the convergence of SGD on a single machine.

Motivation & Objective

• Address the challenge of scaling deep neural network training across multiple machines with minimal network overhead.
• Develop a distributed training method that is hardware-agnostic and suitable for large-scale speech recognition workloads.
• Improve convergence speed and stability of stochastic gradient descent (SGD) in both single-machine and distributed settings.
• Enable effective parameter synchronization across distributed workers without excessive communication costs.

Proposed method

• Periodically average model parameters (e.g., every 1–2 minutes) across multiple training machines to synchronize updates.
• Redistribute the averaged parameters to all workers to maintain consistency across the distributed system.
• Introduce an efficient, approximate implementation of Natural Gradient for Stochastic Gradient Descent (NG-SGD) to stabilize and accelerate training.
• Use the NG-SGD approximation to enhance convergence on single machines and improve the effectiveness of parameter averaging in distributed settings.
• Ensure the method remains effective even when each machine trains on different data subsets.
• Design the framework to be hardware-agnostic, minimizing reliance on specific network topologies or communication patterns.

Experimental results

Research questions

• RQ1 Can periodic parameter averaging enable effective distributed training of DNNs with minimal communication overhead?
• RQ2 How does an approximate natural gradient method improve convergence in both single-machine and distributed training?
• RQ3 To what extent does the combination of parameter averaging and NG-SGD outperform standard SGD in distributed settings?
• RQ4 Can this framework maintain performance and stability when scaling across multiple GPU-equipped or multicore machines?
• RQ5 What is the impact of communication frequency (e.g., every 1–2 minutes) on training stability and convergence?

Key findings

• The combination of periodic parameter averaging and approximate NG-SGD enables stable and effective distributed training across multiple machines.
• The approximate NG-SGD method significantly improves convergence speed and stability on single-machine training compared to standard SGD.
• Parameter averaging alone fails to produce good results, but becomes effective when combined with the NG-SGD approximation.
• The framework achieves high scalability with minimal network traffic, making it suitable for heterogeneous or large-scale hardware clusters.
• The method is robust to data sharding across machines, as each worker trains on different data while maintaining model consistency through averaging.
• The approach is hardware-agnostic, enabling deployment across diverse computing environments without performance degradation.