[Paper Review] A Brief Survey and Comparative Study of Recent Development of Pronoun Coreference Resolution

Hongming Zhang, Xinran Zhao+1 more|arXiv (Cornell University)|Sep 27, 2020

Natural Language Processing Techniques51 references4 citations

TL;DR

This survey provides a comprehensive analysis of recent advances in pronoun coreference resolution (PCR), comparing models on standard and challenging benchmarks like the Winograd Schema Challenge. It reveals that even state-of-the-art models struggle with cross-domain generalization and infrequent antecedents, highlighting limitations in commonsense reasoning and robustness despite strong performance on standard metrics.

ABSTRACT

Pronoun Coreference Resolution (PCR) is the task of resolving pronominal expressions to all mentions they refer to. Compared with the general coreference resolution task, the main challenge of PCR is the coreference relation prediction rather than the mention detection. As one important natural language understanding (NLU) component, pronoun resolution is crucial for many downstream tasks and still challenging for existing models, which motivates us to survey existing approaches and think about how to do better. In this survey, we first introduce representative datasets and models for the ordinary pronoun coreference resolution task. Then we focus on recent progress on hard pronoun coreference resolution problems (e.g., Winograd Schema Challenge) to analyze how well current models can understand commonsense. We conduct extensive experiments to show that even though current models are achieving good performance on the standard evaluation set, they are still not ready to be used in real applications (e.g., all SOTA models struggle on correctly resolving pronouns to infrequent objects). All experiment codes are available at https://github.com/HKUST-KnowComp/PCR.

Motivation & Objective

• To survey recent progress in pronoun coreference resolution (PCR) across standard and challenging benchmarks.
• To evaluate how well current models handle commonsense reasoning, especially in hard cases like the Winograd Schema Challenge.
• To identify key limitations in existing models, particularly their poor performance on rare or infrequent antecedents and cross-domain generalization.
• To provide a unified overview of datasets, models, and evaluation protocols for PCR, supporting future research.
• To release code and resources to encourage reproducible research and model improvement in PCR.

Proposed method

• The authors conduct a comparative survey of representative datasets and models for ordinary pronoun coreference resolution, including CoNLL-2012 and recent end-to-end models.
• They evaluate state-of-the-art models on the CoNLL-2012 dataset under two settings: with and without gold mention annotations, to isolate the challenge of pronoun-to-mention prediction.
• They analyze model performance on the Winograd Schema Challenge (WSC) to assess commonsense reasoning, comparing results across pre-trained language models and fine-tuned variants.
• They investigate model sensitivity to hyper-parameters through extensive ablation studies, especially in low-resource or out-of-domain settings.
• They survey specialized PCR tasks such as medical, multilingual, conversational, gender bias, and visual-aware PCR, highlighting domain-specific challenges and datasets.
• All experimental code is released via GitHub to support reproducibility and community-driven model development.

Experimental results

Research questions

• RQ1 How do current state-of-the-art models perform on standard pronoun coreference resolution benchmarks, and what are their limitations in real-world deployment?
• RQ2 To what extent can existing models resolve pronouns using commonsense reasoning, as tested on the Winograd Schema Challenge?
• RQ3 Why do models still struggle with resolving pronouns to infrequent or rare antecedents, even when performance is high on standard metrics?
• RQ4 How sensitive are current models to hyper-parameter choices, and can this affect reliability in practical applications?
• RQ5 What are the key differences in performance across specialized PCR tasks such as medical, conversational, or visual-aware settings?

Key findings

• Even with strong performance on standard evaluation sets, state-of-the-art models fail to generalize to cross-domain settings, particularly when resolving pronouns to infrequent antecedents.
• The End-to-end model achieves an F1 of 0.705 on the CoNLL-2012 dataset without gold mentions, but performance drops significantly on NP-P and NP-NP relations, indicating that pronoun-to-noun-phrase resolution remains the most challenging subtask.
• All SOTA models struggle with resolving pronouns to infrequent objects, suggesting a critical gap in data efficiency and out-of-vocabulary generalization.
• Fine-tuned pre-trained language models achieve near-human performance on the Winograd Schema Challenge but remain highly sensitive to hyper-parameter settings, limiting real-world reliability.
• The visual-aware PCR task shows that 15% of pronouns in dialogues refer to unmentioned visual objects, underscoring the importance of multimodal grounding for robust dialogue understanding.
• Gender bias persists in pre-trained models, with pronouns like 'he' being disproportionately linked to male-associated roles (e.g., doctor), highlighting a need for bias mitigation in NLP systems.