[Paper Review] Deep Residual Output Layers for Neural Language Generation

Nikolaos Pappas, James Henderson|arXiv (Cornell University)|May 14, 2019

Topic Modeling4 citations

TL;DR

This paper proposes Deep Residual Output Layers (DRILL), a deep residual neural network for learning structured output label embeddings in neural language generation. By applying residual connections and dropout between stacked nonlinear layers, DRILL enhances classifier expressivity without increasing rank or dimensionality, achieving state-of-the-art or competitive results on language modeling and machine translation, including surpassing Transformer (base) and matching RNMT+ with fewer parameters.

ABSTRACT

Many tasks, including language generation, benefit from learning the structure of the output space, particularly when the space of output labels is large and the data is sparse. State-of-the-art neural language models indirectly capture the output space structure in their classifier weights since they lack parameter sharing across output labels. Learning shared output label mappings helps, but existing methods have limited expressivity and are prone to overfitting. In this paper, we investigate the usefulness of more powerful shared mappings for output labels, and propose a deep residual output mapping with dropout between layers to better capture the structure of the output space and avoid overfitting. Evaluations on three language generation tasks show that our output label mapping can match or improve state-of-the-art recurrent and self-attention architectures, and suggest that the classifier does not necessarily need to be high-rank to better model natural language if it is better at capturing the structure of the output space.

Motivation & Objective

• To address the challenge of learning meaningful structure in large, sparse output label spaces common in neural language generation.
• To improve generalization and transfer across low-resource and rare output labels by enhancing the expressivity of the output classifier.
• To explore whether increasing classifier power via deeper label mappings—without increasing rank or dimensionality—can surpass state-of-the-art models.
• To reduce overfitting in high-capacity output classifiers through residual connections and dropout, maintaining model efficiency.

Proposed method

• Proposes a deep residual nonlinear mapping from word embeddings to the joint input-output space, replacing the standard linear classifier.
• Uses residual connections that skip one or more layers, allowing gradients to flow more easily and preserving input information.
• Applies dropout between residual blocks to regularize the network and prevent overfitting.
• Maintains the standard dot product and softmax operation for output prediction, preserving compatibility with existing encoder architectures.
• Employs two distinct dropout strategies: standard dropout on hidden representations and residual-dropout to randomly zero residual connections.
• Retains the same input encoder (e.g., RNN, Transformer) while replacing only the output classifier with the DRILL module.

Experimental results

Research questions

• RQ1 Can a deeper, residual-based output label mapping improve performance in neural language generation without increasing the classifier’s rank or dimensionality?
• RQ2 Does learning a more expressive structure in the output space lead to better generalization, especially for low-frequency or rare words?
• RQ3 Can residual connections and dropout in the output layer reduce overfitting and improve generalization compared to shallow or non-residual mappings?
• RQ4 To what extent can a simple, modular output layer replacement outperform complex, high-capacity encoder architectures?

Key findings

• DRILL achieves a test perplexity of 18.3 on the Wikitext-2 language modeling benchmark, outperforming the Transformer (base) model by 0.8 points and matching the performance of the larger Transformer (big) model with significantly fewer parameters.
• On the English-to-German translation task with a 32K BPE vocabulary, DRILL-based Transformer (base) achieves 28.1 BLEU, surpassing the standard Transformer (base) by 0.8 BLEU and outperforming the dual nonlinear mapping method by 0.6 BLEU.
• The model improves prediction performance for low-frequency words, indicating better transfer learning across semantically similar labels through structured output embeddings.
• DRILL matches or exceeds the performance of complex ensemble models like RNMT+ cascaded and multicol, which use deeper decoders and hybrid architectures, despite being a single-model, non-ensemble component.
• The method achieves state-of-the-art results without modifying the input encoder, suggesting that output layer design is a critical, underexplored lever for performance improvement.