[Paper Review] Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings

Gregor Wiedemann, Steffen Remus+2 more|arXiv (Cornell University)|Sep 23, 2019

Natural Language Processing Techniques37 references121 citations

TL;DR

The paper shows that contextualized word embeddings enable effective word sense disambiguation via a simple kNN approach, with BERT achieving new state-of-the-art on lexical sample WSD datasets and revealing sense clustering in embedding space.

ABSTRACT

Contextualized word embeddings (CWE) such as provided by ELMo (Peters et al., 2018), Flair NLP (Akbik et al., 2018), or BERT (Devlin et al., 2019) are a major recent innovation in NLP. CWEs provide semantic vector representations of words depending on their respective context. Their advantage over static word embeddings has been shown for a number of tasks, such as text classification, sequence tagging, or machine translation. Since vectors of the same word type can vary depending on the respective context, they implicitly provide a model for word sense disambiguation (WSD). We introduce a simple but effective approach to WSD using a nearest neighbor classification on CWEs. We compare the performance of different CWE models for the task and can report improvements above the current state of the art for two standard WSD benchmark datasets. We further show that the pre-trained BERT model is able to place polysemic words into distinct 'sense' regions of the embedding space, while ELMo and Flair NLP do not seem to possess this ability.

Motivation & Objective

• Motivate word sense disambiguation as a testbed for contextualized word embeddings (CWEs).
• Evaluate how CWEs encode senses across different models (Flair, ELMo, BERT).
• Test a simple, interpretable kNN-based WSD method to assess proximity-based sense distinctions.
• Investigate dataset effects (SE-2, SE-3, S7-T7, S7-T17) on WSD performance.
• Provide qualitative analysis of when CWEs place senses into separable regions in embedding space.

Proposed method

• Use a simple k-nearest neighbor classifier on CWE vectors to perform WSD (localized kNN using training senses).
• Represent target word by lemma-based vector: average wordpiece CWEs for BERT when targeting a word, or standard CWE vectors for other models.
• For BERT, concatenate the averaged last-four-layer wordpiece vectors of the target token.
• Vary k over a wide range and apply a normalization to handle imbalanced sense frequencies (k' = min(k, number of examples for a sense)).
• Compare three CWE models (Flair, ELMo, BERT) on four standard WSD datasets (SE-2, SE-3, S7-T7, S7-T17) and two training corpora (SemCor, WNGT).
• Provide qualitative analysis including t-SNE visualizations of sense clusters and error analysis.

Experimental results

Research questions

• RQ1 Can contextualized embeddings enable effective WSD when used with a simple non-parametric classifier?
• RQ2 Do different CWE models (Flair, ELMo, BERT) organize senses in embedding space differently?
• RQ3 How does training data sparseness affect kNN-based WSD performance on standard benchmarks?
• RQ4 To what extent can BERT place polysemous senses into separable regions in embedding space compared to ELMo and Flair?
• RQ5 Does restricting to POS-labeled senses improve WSD accuracy for all-words tasks?

Key findings

• BERT-based CWEs achieve the best WSD performance among the three models for SE-2, SE-3, and S7-T7 tasks (with notable improvements over prior state-of-the-art on SE-2 and SE-3).
• kNN WSD shows that sense regions are more separable in BERT embeddings than in ELMo or Flair, evidenced by clustering in visualizations and higher near-neighbor accuracy.
• WSD performance degrades on all-words tasks (S7-T7, S7-T17) due to data sparsity and domain mismatch between training data (SemCor/WNGT) and test data.
• A POS-restricted (lemma+POS) variant improves F1 on S7-T7 and S7-T17, indicating sense disambiguation benefits from constraining morphosyntactic categories in imbalanced datasets.
• Increasing k tends to stabilize results and can yield state-of-the-art results on lexical sample tasks when combined with BERT embeddings.
• Qualitative error analysis reveals near-miss errors often occur due to fine-grained verb senses and cross-POS confusions, but POS restriction helps reduce such errors.