[Paper Review] A Survey on Contextual Embeddings

Qi Liu, Matt J. Kusner+4 more|arXiv (Cornell University)|Mar 16, 2020

Topic Modeling77 references114 citations

TL;DR

This survey provides a comprehensive overview of contextual embeddings, covering their pre-training via language modeling, cross-lingual adaptation, application in downstream NLP tasks, model compression techniques, and analysis of linguistic knowledge encoded in these representations. It highlights state-of-the-art models like BERT, RoBERTa, and ELECTRA, and identifies key challenges in interpretability, robustness, and controlled generation.

ABSTRACT

Contextual embeddings, such as ELMo and BERT, move beyond global word representations like Word2Vec and achieve ground-breaking performance on a wide range of natural language processing tasks. Contextual embeddings assign each word a representation based on its context, thereby capturing uses of words across varied contexts and encoding knowledge that transfers across languages. In this survey, we review existing contextual embedding models, cross-lingual polyglot pre-training, the application of contextual embeddings in downstream tasks, model compression, and model analyses.

Motivation & Objective

• To provide a comprehensive review of contextual embedding models and their evolution in NLP.
• To examine pre-training methods for contextual embeddings, including unsupervised and supervised approaches.
• To analyze how contextual embeddings transfer knowledge across languages and tasks.
• To evaluate model compression techniques for deploying efficient, lightweight versions of large models.
• To investigate the linguistic knowledge encoded in contextual representations through probing and visualization.

Proposed method

• Categorizes contextual embedding models based on architecture (e.g., LSTM, Transformer) and pre-training objectives (e.g., language modeling, masked language modeling).
• Reviews unsupervised pre-training via left-to-right and bidirectional language modeling, including models like ELMo, GPT, and BERT.
• Describes multi-lingual pre-training using objectives like masked language modeling (MLM), next sentence prediction (NSP), and text infilling.
• Details distillation-based compression methods such as DistilBERT and TinyBERT, which transfer knowledge from large teacher models to smaller student models.
• Applies knowledge distillation and quantization (e.g., Q-BERT) to reduce model size and inference cost.
• Employs probe classifiers and visualization techniques (e.g., attention maps, loss landscapes) to analyze linguistic knowledge in representations.

Experimental results

Research questions

• RQ1 How do contextual embeddings capture polysemy and context-dependent word meanings more effectively than static word embeddings?
• RQ2 What pre-training objectives lead to the most effective and transferable contextual representations across NLP tasks?
• RQ3 To what extent do pre-trained models encode syntactic and semantic knowledge, and how can this be measured reliably?
• RQ4 How can large contextual embedding models be compressed without significant performance loss for real-world deployment?
• RQ5 What are the key vulnerabilities and limitations in current contextual embedding models, particularly in robustness and controllability?

Key findings

• Contextual embeddings like BERT and RoBERTa achieve state-of-the-art performance across a wide range of NLP tasks, including text classification, question answering, and summarization.
• Probing studies show that BERT learns linguistic structures in a hierarchical manner—part-of-speech tagging and parsing in earlier layers, coreference and semantic roles in deeper layers.
• DistilBERT achieves ~97% of BERT’s performance on GLUE with 60% fewer parameters, demonstrating effective knowledge distillation.
• Visualization of attention heads reveals high redundancy in BERT, and disabling certain heads can improve performance over the full model.
• BERT’s representations encode fine-grained word senses in a low-dimensional subspace, indicating high representational capacity.
• Despite strong performance, probes often fail to distinguish whether linguistic knowledge is truly encoded or if performance stems from high-dimensional data alone.