[Paper Review] Autoregressive Entity Retrieval

Nicola De Cao, Gautier Izacard+2 more|arXiv (Cornell University)|Oct 2, 2020

Topic Modeling65 references200 citations

TL;DR

GENRE retrieves entities by generating their unique names token-by-token in an autoregressive model, using constrained decoding to output valid entity identifiers, achieving strong results across ED, EL, and document retrieval with substantially smaller memory footprint.

ABSTRACT

Entities are at the center of how we represent and aggregate knowledge. For instance, Encyclopedias such as Wikipedia are structured by entities (e.g., one per Wikipedia article). The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering. Current approaches can be understood as classifiers among atomic labels, one for each entity. Their weight vectors are dense entity representations produced by encoding entity meta information such as their descriptions. This approach has several shortcomings: (i) context and entity affinity is mainly captured through a vector dot product, potentially missing fine-grained interactions; (ii) a large memory footprint is needed to store dense representations when considering large entity sets; (iii) an appropriately hard set of negative data has to be subsampled at training time. In this work, we propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion. This mitigates the aforementioned technical issues since: (i) the autoregressive formulation directly captures relations between context and entity name, effectively cross encoding both; (ii) the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with vocabulary size, not entity count; (iii) the softmax loss is computed without subsampling negative data. We experiment with more than 20 datasets on entity disambiguation, end-to-end entity linking and document retrieval tasks, achieving new state-of-the-art or very competitive results while using a tiny fraction of the memory footprint of competing systems. Finally, we demonstrate that new entities can be added by simply specifying their names. Code and pre-trained models at https://github.com/facebookresearch/GENRE.

Motivation & Objective

• Motivate more flexible entity retrieval beyond atomic-label classifiers by leveraging structured, compositional entity names.
• Propose an autoregressive seq2seq framework (GENRE) to generate entity names conditioned on input context.
• Introduce constrained decoding to generate only valid entity identifiers from a predefined candidate set.
• Demonstrate that GENRE achieves strong performance on ED, EL, and document retrieval while drastically reducing memory usage.
• Show that new entities can be added by simply appending their unambiguous names to the candidate set.

Proposed method

• Use a transformer-based seq2seq model (pre-trained with language modeling objective, e.g., BART) fine-tuned to generate entity names.
• Represent entities by their textual names and score an entity e given input x via an autoregressive product pθ(y|x) over y tokens of the entity name.
• Train with standard seq2seq objective (maximum likelihood with teacher forcing) without negative sampling.
• At inference, perform constrained beam search over a trie of valid entity names to output only entities in the candidate set.
• Apply constrained decoding to ensure generated outputs are valid entity identifiers and to enable efficient exact softmax computation.
• Extend autoregressive decoding to end-to-end entity linking using dynamic, markup-based outputs with a dynamic entity name trie.

Experimental results

Research questions

• RQ1 Can an autoregressive model generate entity names conditioned on input context to perform ED, EL, and document retrieval effectively?
• RQ2 Does constraining decoding to a candidate set via a trie preserve accuracy while enabling efficient decoding at scale?
• RQ3 How does GENRE compare with existing bi-encoder/classifier-based retrievers in terms of accuracy and memory footprint across ED, EL, and DR tasks?
• RQ4 Can new entities be incorporated by simply adding their names to the candidate set without retraining?
• RQ5 What is the impact of training data (e.g., pretraining on BLINK, fine-tuning on domain datasets) on ED/EL/DR performance?

Key findings

• GENRE achieves state-of-the-art or competitive results across over 20 datasets in three task families: ED, EL, and page-level DR.
• GENRE substantially reduces memory footprint (about 20x smaller on average) by indexing entity names instead of dense per-entity vectors.
• Constrained beam search over a trie ensures outputs are valid entity names and enables exact softmax computation without negative sampling.
• Using the structured, compositional space of entity names improves generalization, especially when exact mention-name overlaps are partial or absent.
• New entities can be added by simply appending their unambiguous names to the candidate set without retraining.
• On DR tasks (KILT benchmark), GENRE attains an average improvement of up to 13.7 R-precision points over strong baselines, and is the best or near-best across datasets except Natural Questions.
• In ED, GENRE shows modest gains on in-domain data but larger gains in out-of-domain settings, demonstrating strong cross-domain robustness.
• In EL, GENRE is best in-domain on AIDA and shows substantial improvements on several out-of-domain datasets (e.g., Derczynski, KORE50).
• Ablations indicate that constrained decoding and using a candidate set substantially improve performance compared to unconstrained or candidate-free variants.