[Paper Review] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

Ikuya Yamada, Akari Asai+3 more|arXiv (Cornell University)|Oct 2, 2020

Topic Modeling41 references34 citations

TL;DR

LUKE introduces pretrained contextualized word and entity representations using a transformer that treats words and entities as tokens and employs an entity-aware self-attention mechanism, achieving state-of-the-art results on multiple entity-related tasks.

ABSTRACT

Entity representations are useful in natural language tasks involving entities. In this paper, we propose new pretrained contextualized representations of words and entities based on the bidirectional transformer. The proposed model treats words and entities in a given text as independent tokens, and outputs contextualized representations of them. Our model is trained using a new pretraining task based on the masked language model of BERT. The task involves predicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also propose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores. The proposed model achieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains state-of-the-art results on five well-known datasets: Open Entity (entity typing), TACRED (relation classification), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), and SQuAD 1.1 (extractive question answering). Our source code and pretrained representations are available at https://github.com/studio-ousia/luke.

Motivation & Objective

• Motivate improved modeling of entities in text for downstream entity-related tasks.
• Propose a unified Transformer-based model that jointly represents words and entities as tokens.
• Introduce a new pretraining objective that masks and predicts masked entities in Wikipedia-derived data.
• Develop an entity-aware self-attention mechanism that differentiates token types during attention computation.
• Demonstrate state-of-the-art performance across multiple entity-centric benchmarks.

Proposed method

• Treat words and entities as independent tokens in a unified Transformer architecture.
• Use three input embeddings (token, position, and entity type) to form token representations.
• Pretrain with MLM plus an entity-masking objective over a large Wikipedia-derived corpus.
• Introduce entity-aware self-attention with type-dependent query matrices (Q_w2e, Q_e2w, Q_e2e) for cross-type attention.
• Fine-tune with simple linear classifiers on top of LUKE representations for various tasks.

Experimental results

Research questions

• RQ1 Can joint word/entity token representations within a Transformer improve entity-centric tasks (typing, NER, QA, relation classification)?
• RQ2 Does an entity-aware self-attention mechanism improve reasoning about relationships between entities compared to standard self-attention?
• RQ3 To what extent do entity representations from Wikipedia-annotated data improve downstream performance relative to word-only pretraining?

Key findings

• LUKE achieves state-of-the-art results on five datasets: Open Entity (entity typing), TACRED (relation classification), CoNLL-2003 (NER), ReCoRD (cloze-style QA), and SQuAD 1.1 (extractive QA).
• On Open Entity, LUKE attains 78.2 F1 (vs. RoBERTa 76.2), surpassing prior best by 2.0 F1.
• On TACRED, LUKE attains 72.7 F1 (vs. RoBERTa 71.3), surpassing prior best by 0.7 F1.
• On CoNLL-2003, LUKE achieves 94.3 F1 (state of the art, surpassing prior by ~0.8 F1).
• On ReCoRD, LUKE reaches 91.4 F1 (Dev) / 90.9+ EM, outperforming RoBERTa baselines.
• On SQuAD 1.1, LUKE achieves 95.0 F1 (Dev) / 89.8 EM, exceeding RoBERTa and XLNet baselines.