[Paper Review] Data Noising as Smoothing in Neural Network Language Models

Ziang Xie, Sida I. Wang+5 more|arXiv (Cornell University)|Mar 7, 2017

Topic Modeling23 references173 citations

TL;DR

The paper connects input data noising in neural LMs with smoothing in n-gram models, derives noising schemes analogous to interpolation and Kneser-Ney, and demonstrates regularization benefits in language modeling and machine translation.

ABSTRACT

Data noising is an effective technique for regularizing neural network models. While noising is widely adopted in application domains such as vision and speech, commonly used noising primitives have not been developed for discrete sequence-level settings such as language modeling. In this paper, we derive a connection between input noising in neural network language models and smoothing in $n$-gram models. Using this connection, we draw upon ideas from smoothing to develop effective noising schemes. We demonstrate performance gains when applying the proposed schemes to language modeling and machine translation. Finally, we provide empirical analysis validating the relationship between noising and smoothing.

Motivation & Objective

• Motivate data augmentation for neural sequence models to address data sparsity in language modeling.
• Formalize a connection between input noising and classical n-gram smoothing methods.
• Develop noising schemes that mirror interpolation and advanced smoothing (e.g., Kneser-Ney).

Proposed method

• Model RNN/LSTM language models trained with maximum likelihood.
• Introduce unigram and blank noising schemes for input contexts.
• Show that expected pseudocounts under noise correspond to interpolation between higher- and lower-order models.
• Propose adaptive noising probabilities and smarter proposal distributions inspired by discounting and Kneser-Ney smoothing.
• Extend noising to encoder-decoder/seq2seq settings by noise-ing both input and output sequences.

Experimental results

Research questions

• RQ1 How can input data noising approximate smoothing techniques used in n-gram language models?
• RQ2 Can adaptive noising and better proposal distributions improve regularization for neural sequence models?
• RQ3 Do noising-based schemes translate to gains in language modeling and machine translation?

Key findings

• Unigram noising yields a p(xt|xt-1) that is equivalent to a mixture of higher- and lower-order models (interpolation).
• Blank noising and unigram noising act as interpolation smoothing analogues for n-gram models.
• Adaptive discounting improves noising effectiveness across a range of gamma values.
• Noised models produce softmax distributions closer to lower-order distributions (uniform and unigram frequencies) than unnoised models, validating the smoothing analogy.
• For machine translation, blank, unigram, and bigram Kneser-Ney noising improve performance (BLEU gains) over baseline dropout alone.
• Experiments on Penn Treebank and Text8 show perplexity improvements with bigram KN noising, and MT shows BLEU gains of roughly +0.7 to +1.4 over baselines.