[Paper Review] The Referential Reader: A Recurrent Entity Network for Anaphora Resolution

Fei Liu, Luke Zettlemoyer+1 more|arXiv (Cornell University)|Feb 5, 2019

Topic Modeling19 references4 citations

TL;DR

The paper proposes the Referential Reader, a differentiable recurrent entity network that resolves anaphora incrementally during online text processing by maintaining a fixed-size memory of entity mentions. It uses learnable gates to update or overwrite memory cells, achieving strong performance on the GAP dataset with end-to-end training using both supervised coreference and unsupervised language modeling objectives.

ABSTRACT

We present a new architecture for storing and accessing entity mentions during online text processing. While reading the text, entity references are identified, and may be stored by either updating or overwriting a cell in a fixed-length memory. The update operation implies coreference with the other mentions that are stored in the same cell; the overwrite operation causes these mentions to be forgotten. By encoding the memory operations as differentiable gates, it is possible to train the model end-to-end, using both a supervised anaphora resolution objective as well as a supplementary language modeling objective. Evaluation on a dataset of pronoun-name anaphora demonstrates strong performance with purely incremental text processing.

Motivation & Objective

• Address the limitations of mention-pair models in anaphora resolution, which are computationally expensive and cognitively implausible due to their batch processing nature.
• Develop a model that processes text incrementally, mimicking human-like online reference resolution.
• Enable end-to-end training using both supervised coreference resolution and unsupervised language modeling objectives to improve generalization.
• Integrate a differentiable memory mechanism that supports online updates and overwrites, with salience-based memory retention.
• Demonstrate the model's effectiveness on the GAP dataset for pronoun-name anaphora resolution with incremental processing.

Proposed method

• Represent entities using a fixed-length memory of (key, value, salience) tuples, where keys are query representations and values are entity embeddings.
• Use a pre-recurrent state derived from the input token and prior hidden state to control memory operations via differentiable gates.
• Compute entity mention detection using a sigmoid gate on the pre-recurrent state, determining whether a token is a candidate entity reference.
• Compute reference compatibility via a query vector derived from the pre-recurrent state, attending over memory keys to identify potential antecedents.
• Apply update and overwrite gates as continuous, differentiable functions: update gates are attention scores weighted by reference probability, and overwrite gates are learned decisions to replace memory content.
• Integrate the memory state into the GRU hidden state update via a dynamic gate that balances prior hidden state and memory contributions.

Experimental results

Research questions

• RQ1 Can a differentiable, end-to-end trainable memory network achieve strong performance in incremental anaphora resolution?
• RQ2 To what extent can unsupervised language modeling pretraining improve coreference resolution performance in a semi-supervised setting?
• RQ3 How well does the model handle coreference chains and cataphoric references in real-time processing?
• RQ4 Does the memory salience mechanism effectively prioritize salient entities and reduce overwriting of relevant antecedents?
• RQ5 Can the model learn syntactic salience patterns (e.g., subject vs. prepositional phrase) without explicit linguistic supervision?

Key findings

• The Referential Reader achieves an F1 score of 71.4 on the GAP test set when jointly trained with a language modeling objective, outperforming the supervised-only baseline.
• The model demonstrates strong incremental processing behavior, resolving references on-the-fly without requiring full document context.
• The inclusion of a language modeling objective significantly improves performance, reducing the drop in F1 when using only the coreference objective.
• The model learns to ignore low-salience mentions (e.g., names in prepositional phrases) and prioritize pronouns and high-salience entities, aligning with centering theory principles.
• Visualization shows that the model correctly forms coreference links even in complex cases, such as cataphoric references (e.g., 'his' referring to 'Avant' later in the sentence).
• The model’s memory salience mechanism effectively retains relevant entities over long spans, reducing spurious overwrites and improving long-range coreference resolution.