[Paper Review] How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations

Betty van Aken, Benjamin Winter+2 more|arXiv (Cornell University)|Sep 11, 2019

Topic Modeling37 references42 citations

TL;DR

The paper analyzes BERT’s hidden states across layers to understand how QA reasoning emerges, showing phase-like transformations and that fine-tuning has limited impact on semantic abilities.

ABSTRACT

Bidirectional Encoder Representations from Transformers (BERT) reach state-of-the-art results in a variety of Natural Language Processing tasks. However, understanding of their internal functioning is still insufficient and unsatisfactory. In order to better understand BERT and other Transformer-based models, we present a layer-wise analysis of BERT's hidden states. Unlike previous research, which mainly focuses on explaining Transformer models by their attention weights, we argue that hidden states contain equally valuable information. Specifically, our analysis focuses on models fine-tuned on the task of Question Answering (QA) as an example of a complex downstream task. We inspect how QA models transform token vectors in order to find the correct answer. To this end, we apply a set of general and QA-specific probing tasks that reveal the information stored in each representation layer. Our qualitative analysis of hidden state visualizations provides additional insights into BERT's reasoning process. Our results show that the transformations within BERT go through phases that are related to traditional pipeline tasks. The system can therefore implicitly incorporate task-specific information into its token representations. Furthermore, our analysis reveals that fine-tuning has little impact on the models' semantic abilities and that prediction errors can be recognized in the vector representations of even early layers.

Motivation & Objective

• Investigate how BERT transforms token representations across layers during a QA task.
• Reveal whether different layers perform distinct reasoning steps in QA.
• Assess how fine-tuning for QA affects internal layer representations.
• Provide tools to identify why and where QA predictions fail using hidden states.

Proposed method

• Perform layer-wise qualitative visualization of token vectors using dimensionality reduction and clustering.
• Apply a set of probing tasks (Named Entity Labeling, Coreference Resolution, Relation Classification) plus QA-specific tasks (Question Type Classification, Supporting Fact Identification) to each layer.
• Use Edge Probing to convert core NLP tasks into layer-wise classification problems.
• Compare fine-tuned BERT models on SQuAD, HotpotQA, and bAbI with pre-trained counterparts.
• Incorporate GPT-2 as a qualitative comparison to assess transferability of insights to other Transformers.

Experimental results

Research questions

• RQ1 Do Transformer models answer questions decompositionally like humans?
• RQ2 Do specific layers in a multi-layer Transformer solve different tasks during QA?
• RQ3 How does fine-tuning influence the network’s inner state and layer functionalities?
• RQ4 Can layer-wise analysis help explain why a model fails to predict the correct answer?

Key findings

• Hidden states contain interpretable information that progresses through phases similar to traditional NLP tasks.
• Early layers encode general language properties; later layers incorporate task-specific information for QA.
• Named Entity Labeling improves earliest; Coreference Resolution and Relation Classification peak in later layers; Question Type information is task-dependent.
• Question and Supporting Facts are matched in vector space across layers, with higher layers aligning relevant tokens for answer extraction.
• Fine-tuning has limited impact on core NLP abilities; most information is already present pre-training, with small weight changes during fine-tuning.
• Positional embeddings maintain their influence into later layers, supporting sequential information in Transformer representations.