[Paper Review] End-to-end Multilingual Coreference Resolution with Mention Head Prediction

Ondřej Pražák, Miloslav Konopík|arXiv (Cornell University)|Sep 26, 2022

Topic Modeling4 citations

TL;DR

This paper presents an end-to-end multilingual coreference resolution model that improves performance through mention head prediction, leveraging XLM-RoBERTa-large with joint multilingual pretraining. The system achieves third place in the CRAC 2022 shared task and ranks first on two datasets by directly optimizing for the CoNLL F1 score using head-word spans, demonstrating that head prediction enhances alignment with the official evaluation metric.

ABSTRACT

This paper describes our approach to the CRAC 2022 Shared Task on Multilingual Coreference Resolution. Our model is based on a state-of-the-art end-to-end coreference resolution system. Apart from joined multilingual training, we improved our results with mention head prediction. We also tried to integrate dependency information into our model. Our system ended up in $3^{rd}$ place. Moreover, we reached the best performance on two datasets out of 13.

Motivation & Objective

• To address multilingual coreference resolution in a unified, end-to-end framework using a single model across 13 languages.
• To improve performance on the CRAC 2022 shared task by aligning model predictions with the official CoNLL F1 scorer, which uses minimal mention spans centered on headwords.
• To investigate whether predicting mention heads instead of full spans enhances model alignment with the evaluation metric.
• To explore the integration of syntactic structure (dependency trees) into the model for improved coreference resolution.
• To evaluate the effectiveness of joint multilingual pretraining versus monolingual fine-tuning on low-resource languages.

Proposed method

• The model is based on a transformer-based end-to-end coreference resolution framework that predicts antecedents directly from all possible mention spans without prior mention detection.
• The system uses XLM-RoBERTa-large as the base encoder, with joint multilingual pretraining on all 13 CorefUD datasets before fine-tuning per language.
• A dedicated head prediction head is added on top of the mention representation to predict the headword of each mention, optimizing for the minimal span used in the CoNLL scorer.
• Mentions are represented by their full spans during clustering, but only the predicted head is used for evaluation to match the scorer’s partial mention matching rule.
• Syntactic dependency information was attempted to be injected via dependency trees, but this did not improve performance due to implementation issues.
• The model maximizes the marginal log-likelihood of correct antecedents during training, defined as $ J(D) = ext{log} ig( ext{prod}_{i=1}^{N} ext{sum}_{ ext{y} ext{ in } Y(i) ext{ and GOLD}(i)} P( ext{y}) ig) $.

Experimental results

Research questions

• RQ1 Does joint multilingual pretraining improve coreference resolution performance across low-resource and high-resource languages in a shared task setting?
• RQ2 To what extent does predicting only the mention head rather than the full span improve alignment with the CoNLL F1 scorer that uses minimal spans?
• RQ3 Can syntactic structure from dependency trees be effectively integrated into an end-to-end coreference model to boost performance?
• RQ4 Why does the monolingual model outperform the multilingual model for Polish, despite the latter’s larger parameter count?
• RQ5 How does the model’s performance vary across languages with differing proportions of singletons and discontinuous mentions?

Key findings

• The system achieved third place overall in the CRAC 2022 shared task, with a mean CoNLL F1 score of 65.94.
• The model achieved the highest performance on two datasets—German-ParCorFull (73.90 F1) and Hungarian-SzegedKoref (65.01 F1)—surpassing all other systems.
• Joint multilingual pretraining significantly improved performance on smaller datasets (e.g., German), where fine-tuning from random initialization would be infeasible.
• Mention head prediction led to a measurable performance gain on three datasets, indicating that aligning with the scorer’s minimal span rule improves evaluation scores.
• The integration of dependency tree information did not improve results, likely due to implementation bugs, despite theoretical potential for syntactic enhancement.
• The monolingual model for Polish outperformed the multilingual XLM-RoBERTa-large model, attributed to the availability of a large, high-quality monolingual model and the dataset’s size.