[Paper Review] Deepening Hidden Representations from Pre-trained Language Models

Junjie Yang, Hai Zhao|arXiv (Cornell University)|Nov 5, 2019

Topic Modeling36 references4 citations

TL;DR

This paper proposes HIdden Representation Extractor (HIRE), a dynamic mechanism that enhances pre-trained language models by fusing complementary representations from intermediate Transformer layers with the final layer's output. Using RoBERTa as a backbone, HIRE improves performance across GLUE tasks, achieving state-of-the-art results on SST-2 and outperforming the baseline on 5 out of 9 tasks by adaptively weighting hidden states based on task and input-specific needs.

ABSTRACT

Transformer-based pre-trained language models have proven to be effective for learning contextualized language representation. However, current approaches only take advantage of the output of the encoder's final layer when fine-tuning the downstream tasks. We argue that only taking single layer's output restricts the power of pre-trained representation. Thus we deepen the representation learned by the model by fusing the hidden representation in terms of an explicit HIdden Representation Extractor (HIRE), which automatically absorbs the complementary representation with respect to the output from the final layer. Utilizing RoBERTa as the backbone encoder, our proposed improvement over the pre-trained models is shown effective on multiple natural language understanding tasks and help our model rival with the state-of-the-art models on the GLUE benchmark.

Motivation & Objective

• To address the limitation of relying solely on the final layer's output in fine-tuned pre-trained language models.
• To explore whether intermediate layers contain complementary information not captured by the final layer.
• To design a dynamic, adaptive mechanism that fuses multi-layer representations to improve contextualized language representation.
• To verify that the proposed method enhances performance across diverse NLU tasks without modifying the backbone model's architecture or pre-training.
• To analyze how representation importance varies across layers, tasks, and individual examples.

Proposed method

• HIRE is a learnable network that extracts complementary representations from all hidden layers except the final one.
• The method uses a dynamic importance scoring mechanism that assigns adaptive weights to each hidden layer based on input and task context.
• A fusion network combines the final layer's output with the HIRE-extracted representation using a gated recurrent unit (GRU) architecture.
• The importance scores are computed via a differentiable soft attention mechanism that aggregates contributions from all intermediate layers.
• The model is trained end-to-end with the same objective as the original RoBERTa, preserving the backbone's pre-training and architecture.
• The fusion process integrates the refined representation into the downstream task head without altering the original model's structure.

Experimental results

Research questions

• RQ1 Can intermediate layers of pre-trained Transformer models provide complementary information not captured by the final layer?
• RQ2 How does the contribution of each hidden layer vary across different NLU tasks and individual input examples?
• RQ3 Can a dynamic, adaptive mechanism for combining multi-layer representations improve downstream performance compared to fixed or single-layer fusion?
• RQ4 Does the proposed HIRE component enhance performance on diverse GLUE benchmark tasks without retraining the backbone model?
• RQ5 What is the optimal number of GRU layers in the fusion network for balancing performance and complexity?

Key findings

• HIRE improved performance on 5 out of 9 tasks in the GLUE benchmark, with the most significant gain on the SST-2 sentiment classification task.
• The model achieved state-of-the-art results on the SST-2 dataset, outperforming the baseline RoBERTa model.
• The dynamic importance scoring mechanism outperformed fixed strategies such as mean or random weighting across layers, demonstrating the value of adaptivity.
• Ablation studies showed that using two GRU layers in the fusion network yielded the best performance on CoLA, with 69.7 Matthews correlation.
• Layer importance distributions varied significantly across tasks: middle and lower layers contributed most for single-sentence and similarity tasks, while all layers contributed nearly equally for STS-B and RTE.
• Visualization confirmed that HIRE adapts not only across tasks but also across individual examples, with consistent patterns such as layers 21 and 22 being most influential in SST-2.