[Paper Review] Contextual Temperature for Language Modeling

Pei-Hsin Wang, Sheng-Iou Hsieh+5 more|arXiv (Cornell University)|Dec 25, 2020

Topic Modeling24 references4 citations

TL;DR

This paper proposes contextual temperature, a method that learns a unique temperature for each vocabulary token based on its contextual history, enabling dynamic uncertainty control in language modeling. By optimizing temperature per token and context, the approach achieves state-of-the-art results with a perplexity of 55.31 on Penn Treebank and 62.89 on WikiText-2, significantly outperforming fixed or schedule-based temperature scaling.

ABSTRACT

Temperature scaling has been widely used as an effective approach to control the smoothness of a distribution, which helps the model performance in various tasks. Current practices to apply temperature scaling assume either a fixed, or a manually-crafted dynamically changing schedule. However, our studies indicate that the individual optimal trajectory for each class can change with the context. To this end, we propose contextual temperature, a generalized approach that learns an optimal temperature trajectory for each vocabulary over the context. Experimental results confirm that the proposed method significantly improves state-of-the-art language models, achieving a perplexity of 55.31 and 62.89 on the test set of Penn Treebank and WikiText-2, respectively. In-depth analyses show that the behaviour of the learned temperature schedules varies dramatically by vocabulary, and that the optimal schedules help in controlling the uncertainties. These evidences further justify the need for the proposed method and its advantages over fixed temperature schedules.

Motivation & Objective

• To address the limitation of fixed or handcrafted temperature schedules in controlling model uncertainty during language modeling.
• To investigate whether optimal temperature trajectories vary by vocabulary token and context.
• To develop a generalized temperature scaling mechanism that adapts per token and context to improve model calibration and performance.
• To demonstrate that dynamic temperature control enhances sequence modeling by reducing uncertainty in early sequence positions and suppressing it later.

Proposed method

• The method introduces a learnable temperature vector for each token, where the temperature is conditioned on the hidden state of the transformer encoder.
• A temperature network, parameterized by a small feed-forward network, predicts the temperature for each token based on the context embedding.
• The temperature is applied to the logits before the Softmax function, modifying the output distribution: $ p_i = \frac{\exp(z_i / \tau_i)}{\sum_j \exp(z_j / \tau_j)} $.
• The temperature network is trained end-to-end with the language model using cross-entropy loss, allowing gradients to flow back to adjust per-token temperature.
• The temperature values are normalized via a Softmax-like operation over a learned temperature vector to ensure stability and differentiability.
• The method is applied in a mixture-of-experts (MoS) setting, referred to as CT-MoS, to further improve performance on long-context sequences.

Experimental results

Research questions

• RQ1 Does the optimal temperature schedule for a given token vary significantly with context?
• RQ2 Can learning a context-dependent temperature per token improve language model perplexity and calibration?
• RQ3 How does dynamic temperature control affect uncertainty distribution across different positions in a sequence?
• RQ4 Can per-token temperature adaptation outperform fixed or global temperature schedules in sequence modeling?

Key findings

• The proposed contextual temperature method achieves a test perplexity of 55.31 on the Penn Treebank dataset, representing a significant improvement over baseline models.
• On the WikiText-2 dataset, the method achieves a perplexity of 62.89, demonstrating consistent gains across multiple benchmarks.
• Analysis shows that the average temperature is higher at the beginning of sentences and gradually decreases toward the end, indicating increased stochasticity at the start and reduced uncertainty later.
• Per-token temperature schedules vary dramatically: some tokens exhibit 'heating-up' behavior, while others show steady cooling, reflecting context-specific uncertainty control.
• Ablation studies confirm that the dynamic, context-aware temperature scheduling leads to better performance than fixed or global temperature scaling.
• Case studies show that contextual temperature improves token selection—e.g., correctly predicting 'results' or 'loss' over 'treasury' or '$'—by adjusting probability scaling via learned temperature.