[Paper Review] Universal Text Representation from BERT: An Empirical Study

Xiaofei Ma, Zhiguo Wang+3 more|arXiv (Cornell University)|Oct 17, 2019

Topic Modeling17 references40 citations

TL;DR

The paper empirically analyzes BERT activations as universal text embeddings for sentence and passage representations, showing layer choice, pooling, and fine-tuning impact performance across tasks, with notable gaps to fully supervised in-domain BERT on QA.

ABSTRACT

We present a systematic investigation of layer-wise BERT activations for general-purpose text representations to understand what linguistic information they capture and how transferable they are across different tasks. Sentence-level embeddings are evaluated against two state-of-the-art models on downstream and probing tasks from SentEval, while passage-level embeddings are evaluated on four question-answering (QA) datasets under a learning-to-rank problem setting. Embeddings from the pre-trained BERT model perform poorly in semantic similarity and sentence surface information probing tasks. Fine-tuning BERT on natural language inference data greatly improves the quality of the embeddings. Combining embeddings from different BERT layers can further boost performance. BERT embeddings outperform BM25 baseline significantly on factoid QA datasets at the passage level, but fail to perform better than BM25 on non-factoid datasets. For all QA datasets, there is a gap between embedding-based method and in-domain fine-tuned BERT (we report new state-of-the-art results on two datasets), which suggests deep interactions between question and answer pairs are critical for those hard tasks.

Motivation & Objective

• Investigate what linguistic information BERT layer activations capture when used as universal text embeddings.
• Evaluate sentence-level embeddings on downstream and probing tasks and compare with state-of-the-art models.
• Evaluate passage-level embeddings on QA datasets under learning-to-rank settings to assess transferability.
• Determine effective pooling strategies and layer combinations for optimal embedding performance.

Proposed method

• Extract embeddings from different BERT encoder layers and apply a logistic regression head to assess task performance.
• Evaluate pooling methods (CLS, SEP, mean, max) to determine which yields best sentence representations.
• Fine-tune BERT on natural language inference data (MNLI, SNLI) to study impact on embeddings.
• Combine embeddings from multiple layers and assess simple readouts (logistic regression, MLP) to test if linear separability suffices.
• Evaluate passage embeddings on four QA datasets using a learning-to-rank setup with various interaction schemes between query and passage embeddings.

Experimental results

Research questions

• RQ1 How well do BERT activations serve as universal sentence embeddings across benchmark tasks?
• RQ2 How do layer depth and pooling strategies affect the quality of BERT-based embeddings for linguistic information (syntactic, semantic, surface)?
• RQ3 What is the impact of pre-trained versus fine-tuned BERT on embedding quality for sentence tasks?
• RQ4 Can combining embeddings from multiple layers improve performance, and is a simple linear readout sufficient?
• RQ5 How do BERT embeddings perform for passage-level QA under learning-to-rank, compared to BM25 and fully fine-tuned BERT?

Key findings

• Top-layer embeddings generally outperform lower-layer ones, but middle layers excel on certain semantic probing tasks.
• Mean-pooling of BERT hidden states consistently outperforms CLS, SEP, and max pooling across task categories.
• Fine-tuning BERT on NLI data improves embedding quality, especially for semantic similarity and entailment tasks.
• Combining embeddings from the top and bottom layers gives the best sentence-level performance, with limited gains from adding multiple layers or moving beyond a logistic regression readout.
• For passage QA, in-domain fine-tuned BERT achieves the best results; BERT embeddings outperform BM25 on factoid datasets but underperform on non-factoid datasets; multi-layer pooling and interaction schemes enhance performance, with in-domain fine-tuning providing the strongest gains.