[Paper Review] Hybrid Data-Model Parallel Training for Sequence-to-Sequence Recurrent Neural Network Machine Translation

Junya Ono, Masao Utiyama+1 more|arXiv (Cornell University)|Aug 1, 2019

Natural Language Processing Techniques19 references1 citations

TL;DR

This paper proposes a hybrid data-model parallel training approach for sequence-to-sequence RNNs in machine translation, using model parallelism for the RNN encoder-decoder and data parallelism for the attention-softmax components. The method achieves a 4.13–4.20× speedup on 4 GPUs without degrading BLEU scores, enabling faster training for tasks like patent translation.

ABSTRACT

Reduction of training time is an important issue in many tasks like patent translation involving neural networks. Data parallelism and model parallelism are two common approaches for reducing training time using multiple graphics processing units (GPUs) on one machine. In this paper, we propose a hybrid data-model parallel approach for sequence-to-sequence (Seq2Seq) recurrent neural network (RNN) machine translation. We apply a model parallel approach to the RNN encoder-decoder part of the Seq2Seq model and a data parallel approach to the attention-softmax part of the model. We achieved a speed-up of 4.13 to 4.20 times when using 4 GPUs compared with the training speed when using 1 GPU without affecting machine translation accuracy as measured in terms of BLEU scores.

Motivation & Objective

• Reduce training time for sequence-to-sequence RNN models in machine translation, particularly for compute-intensive tasks like patent translation.
• Address the scalability limitations of pure data or model parallelism by combining both strategies.
• Maintain high translation accuracy, measured by BLEU scores, while accelerating training on multi-GPU systems.
• Optimize GPU utilization by assigning different model components to different parallelization strategies based on computational and memory characteristics.

Proposed method

• Apply model parallelism to the RNN encoder-decoder components, distributing hidden state computations across GPUs.
• Use data parallelism for the attention-softmax components, splitting the batch of sequences across GPUs for independent processing.
• Coordinate communication between GPU partitions using parameter synchronization and gradient averaging during backpropagation.
• Balance computational load by assigning memory-intensive RNN layers to model parallelism and compute-intensive attention mechanisms to data parallelism.
• Implement efficient communication protocols between GPU partitions to minimize overhead in the hybrid training pipeline.

Experimental results

Research questions

• RQ1 Can a hybrid data-model parallel approach reduce training time in Seq2Seq RNNs without sacrificing translation quality?
• RQ2 How does the combination of model and data parallelism compare to pure parallelization strategies in terms of speedup and accuracy?
• RQ3 What is the optimal partitioning strategy for RNN encoder-decoder and attention-softmax components to maximize GPU utilization?
• RQ4 To what extent does the hybrid approach preserve BLEU scores compared to single-GPU training?

Key findings

• The hybrid approach achieved a 4.13 to 4.20 times speedup when training with 4 GPUs compared to a single GPU.
• The speedup was achieved without any degradation in translation quality, as confirmed by consistent BLEU scores across training runs.
• The method effectively balances computational load by leveraging model parallelism for RNNs and data parallelism for attention-softmax components.
• The communication overhead between GPUs was minimized through efficient synchronization, maintaining high training efficiency.