[Paper Review] Global Entity Disambiguation with Pretrained Contextualized Embeddings of Words and Entities

Ikuya Yamada, Koki Washio+2 more|arXiv (Cornell University)|Jan 1, 2020

Topic Modeling30 references29 citations

TL;DR

This paper proposes a global entity disambiguation model leveraging BERT-based contextualized embeddings for both words and entities, trained via a masked entity prediction task on Wikipedia-annotated texts. By framing ED as a sequential decision task to capture global context, the model achieves state-of-the-art performance on six standard datasets, outperforming prior methods on all but one.

ABSTRACT

We propose a new global entity disambiguation (ED) model based on contextualized embeddings of words and entities. Our model is based on a bidirectional transformer encoder (i.e., BERT) and produces contextualized embeddings for words and entities in the input text. The model is trained using a new masked entity prediction task that aims to train the model by predicting randomly masked entities in entity-annotated texts obtained from Wikipedia. We further extend the model by solving ED as a sequential decision task to capture global contextual information. We evaluate our model using six standard ED datasets and achieve new state-of-the-art results on all but one dataset.

Motivation & Objective

• To improve global entity disambiguation by leveraging contextualized word and entity embeddings from a bidirectional transformer encoder.
• To address the limitations of local context modeling in existing ED approaches by incorporating global contextual information through sequential decision making.
• To train the model effectively on entity-annotated texts using a novel masked entity prediction objective.
• To achieve state-of-the-art performance on standard entity disambiguation benchmarks.

Proposed method

• Utilizes a bidirectional transformer encoder (BERT) to generate contextualized embeddings for both words and entities in the input text.
• Introduces a masked entity prediction pretraining task, where randomly masked entities in Wikipedia-annotated texts are predicted by the model.
• Extends the model to treat entity disambiguation as a sequential decision task, enabling the use of global context across the entire document.
• Leverages the contextualized representations from the transformer encoder to jointly model word and entity semantics during inference.
• Fine-tunes the model end-to-end on downstream entity disambiguation datasets using standard evaluation protocols.
• Employs a conditional probability model over entity candidates that integrates contextual features from the entire input sequence.

Experimental results

Research questions

• RQ1 Can a BERT-based model with joint word and entity contextualization improve entity disambiguation performance compared to prior methods?
• RQ2 Does a masked entity prediction pretraining objective effectively capture the relationships between words and entities in context?
• RQ3 To what extent does modeling entity disambiguation as a sequential decision task enhance performance by leveraging global context?
• RQ4 How does the proposed model compare to existing state-of-the-art approaches across diverse entity disambiguation benchmarks?

Key findings

• The proposed model achieves state-of-the-art results on six standard entity disambiguation datasets, demonstrating consistent performance gains.
• The model outperforms previous methods on all but one of the six benchmark datasets, indicating broad effectiveness across diverse domains.
• The masked entity prediction pretraining objective effectively improves the model's ability to predict correct entities in context.
• Formulating ED as a sequential decision task significantly improves performance by enabling the model to consider long-range dependencies and global context.
• The integration of contextualized word and entity embeddings leads to more accurate entity predictions than models relying solely on word-level context.
• The model demonstrates strong generalization across different types of entity-annotated texts, particularly in long-context and ambiguous scenarios.