[Paper Review] Dual Skew Divergence Loss for Neural Machine Translation

Fengshun Xiao, Yingting Wu+3 more|arXiv (Cornell University)|Sep 27, 2018

Natural Language Processing Techniques34 references4 citations

TL;DR

This paper proposes Dual Skew Divergence (DSD) loss, an extended training objective for neural machine translation that improves generalization and avoids local optima by balancing error avoidance and robustness. Empirical results on WMT 2014 English-German and English-French benchmarks show DSD yields superior translation performance over maximum likelihood and baseline losses.

ABSTRACT

For neural sequence model training, maximum likelihood (ML) has been commonly adopted to optimize model parameters with respect to the corresponding objective. However, in the case of sequence prediction tasks like neural machine translation (NMT), training with the ML-based cross entropy loss would often lead to models that overgeneralize and plunge into local optima. In this paper, we propose an extended loss function called dual skew divergence (DSD), which aims to give a better tradeoff between generalization ability and error avoidance during NMT training. Our empirical study indicates that switching to DSD loss after the convergence of ML training helps the model skip the local optimum and stimulates a stable performance improvement. The evaluations on WMT 2014 English-German and English-French translation tasks demonstrate that the proposed loss indeed helps bring about better translation performance than several baselines.

Motivation & Objective

• To address the limitations of maximum likelihood (ML) training in neural machine translation, which often leads to overgeneralization and convergence to poor local optima.
• To develop a loss function that better balances model generalization and error avoidance during NMT training.
• To improve translation performance by enabling models to escape suboptimal solutions after initial ML convergence.

Proposed method

• Propose Dual Skew Divergence (DSD) as an extended loss function that combines two skew divergences to better align model predictions with reference outputs.
• Apply DSD loss after the model converges under maximum likelihood training to refine parameters and escape local optima.
• Use the skew divergence measure to penalize both overconfidence and underconfidence in predictions, enhancing robustness.
• Integrate DSD as a fine-tuning objective following standard ML training, leveraging its ability to correct distributional mismatches.
• Design the loss to be differentiable and compatible with standard NMT architectures and training pipelines.

Experimental results

Research questions

• RQ1 Can a modified loss function improve translation performance beyond standard maximum likelihood training in NMT?
• RQ2 Does Dual Skew Divergence effectively help models escape local optima during training?
• RQ3 How does DSD compare to baseline loss functions in terms of generalization and robustness on standard NMT benchmarks?

Key findings

• Switching to DSD loss after ML convergence leads to stable performance improvements on both English-German and English-French translation tasks.
• The proposed DSD loss achieves better translation performance than maximum likelihood and other baseline loss functions on WMT 2014 benchmarks.
• DSD effectively mitigates overgeneralization and helps models avoid suboptimal local optima during training.
• Empirical results confirm that DSD enhances model robustness by balancing error avoidance and generalization capacity.