[Paper Review] Improving Context Aware Language Models

Aaron Jaech, Mari Ostendorf|arXiv (Cornell University)|Apr 21, 2017

Topic Modeling26 references3 citations

TL;DR

This paper proposes a novel method to improve context-aware language models by enhancing both the hidden and output layers of RNN language models using multiplicative rescaling and feature hashing for context-dependent adaptation. The approach outperforms standard concatenation-based adaptation across three corpora, showing consistent improvements in perplexity and classification accuracy, especially when combining low-rank and hashed bias terms for the output layer.

ABSTRACT

Increased adaptability of RNN language models leads to improved predictions that benefit many applications. However, current methods do not take full advantage of the RNN structure. We show that the most widely-used approach to adaptation (concatenating the context with the word embedding at the input to the recurrent layer) is outperformed by a model that has some low-cost improvements: adaptation of both the hidden and output layers. and a feature hashing bias term to capture context idiosyncrasies. Experiments on language modeling and classification tasks using three different corpora demonstrate the advantages of the proposed techniques.

Motivation & Objective

• To address the limitations of standard context adaptation in RNN language models, which rely on simple concatenation of context embeddings and fail to exploit the full expressiveness of RNN structures.
• To improve model adaptability by introducing multiplicative rescaling of hidden layer weights based on context, enabling more flexible parameter adaptation than additive bias shifts alone.
• To reduce memory costs of context-dependent output layer adaptation while preserving sensitivity to rare or idiosyncratic context-specific patterns through feature hashing with Bloom filters.
• To evaluate the combined effect of hidden and output layer adaptations on diverse language modeling and classification tasks across multiple corpora.

Proposed method

• Adapt the hidden layer using additive context adaptation via a learned context projection matrix F, which adds a context-dependent bias term to the hidden state computation.
• Introduce multiplicative rescaling of the input and recurrent weight matrices (U and S) using context-dependent scaling vectors Cu and Cw, enabling dynamic, non-linear adaptation of hidden state transitions.
• Model the output layer using a low-rank approximation of the output weight matrix V, combined with a learned hash-based correction term to capture context-specific deviations.
• Use feature hashing with Bloom filters to efficiently represent context-dependent bias corrections, reducing memory cost while minimizing hash collisions.
• Combine the low-rank output layer with the hashed bias term to form a hybrid output adaptation mechanism that balances expressiveness and efficiency.
• Train the model end-to-end using cross-entropy loss, with context embeddings derived from a multi-layer perceptron combining multiple context variables (e.g., speaker, role, topic).

Experimental results

Research questions

• RQ1 Does multiplicative rescaling of the hidden layer weights improve performance compared to standard additive context adaptation in RNN language models?
• RQ2 Can a hybrid output layer combining low-rank approximation and feature-hashed bias terms outperform standard low-rank output adaptation in terms of perplexity and classification accuracy?
• RQ3 How does the performance of the proposed model vary across different types of context variables, such as speaker identity or topic, in real-world text data?
• RQ4 Is there a disconnect between perplexity improvements and gains in downstream classification tasks, and if so, what does this imply for model design?

Key findings

• The proposed model with multiplicative rescaling of the hidden layer and hybrid output layer adaptation consistently outperformed the standard method (concatenated context at input only) on all three corpora: SCOTUS, Reddit, and Twitter.
• On the SCOTUS dataset, multiplicative hidden layer adaptation reduced perplexity by 1.8% compared to the baseline, while the hybrid output layer improved perplexity by 2.4%.
• For text classification tasks, the full model with both hidden and output layer adaptations achieved a 3.1% absolute improvement in F1 score over the baseline on the SCOTUS dataset.
• The combination of low-rank and hashed bias terms for the output layer provided consistent gains in perplexity across all datasets, even when the vocabulary and number of contexts were large.
• The model demonstrated strong generation quality, with beam search outputs showing context-specific linguistic patterns—e.g., Justice Breyer frequently starting with "I mean", and Justice Kagan with "Well"—indicating effective context modeling.
• Context variables that are predictable from the text itself (e.g., sentiment in tweets) provided little benefit, suggesting that external, non-redundant context features are more effective.