[Paper Review] UNITER-Based Situated Coreference Resolution with Rich Multimodal Input

Yichen Huang, Yuchen Wang+1 more|arXiv (Cornell University)|Dec 7, 2021

Speech and dialogue systems4 citations

TL;DR

This paper proposes a UNITER-based model for multimodal coreference resolution in the SIMMC 2.0 dataset, leveraging rich inputs including dialog history, visual features, knowledge base entries, object coordinates, and scene graphs. The approach achieves an object F1 score of 73.3% on the test-std set, ranking second in the DSTC10 challenge, with a 40.7 percentage point improvement over the baseline.

ABSTRACT

We present our work on the multimodal coreference resolution task of the Situated and Interactive Multimodal Conversation 2.0 (SIMMC 2.0) dataset as a part of the tenth Dialog System Technology Challenge (DSTC10). We propose a UNITER-based model utilizing rich multimodal context such as textual dialog history, object knowledge base and visual dialog scenes to determine whether each object in the current scene is mentioned in the current dialog turn. Results show that the proposed approach outperforms the official DSTC10 baseline substantially, with the object F1 score boosted from 36.6% to 77.3% on the development set, demonstrating the effectiveness of the proposed object representations from rich multimodal input. Our model ranks second in the official evaluation on the object coreference resolution task with an F1 score of 73.3% after model ensembling.

Motivation & Objective

• To address the challenge of coreference resolution in multimodal, situated dialog systems where references must be resolved across vision, language, and knowledge.
• To improve upon the GPT-2 baseline in the DSTC10 multimodal coreference resolution task by leveraging structured multimodal inputs.
• To investigate the effectiveness of diverse multimodal features—such as object coordinates, KB embeddings, and scene graphs—on coreference resolution performance.
• To develop a robust, end-to-end framework that treats each object as a candidate for mention, using a binary classification head per object.

Proposed method

• The model uses UNITER to jointly encode multimodal inputs, including dialog history, visual features from a pre-trained CNN, and object-specific embeddings from knowledge base entries.
• Each object is represented via a composite embedding combining object index, image features from Faster R-CNN or CLIP, KB descriptions via BERT/SBERT, and auxiliary features like object position and prior mention status.
• Scene graph relationships are incorporated either as attention bias or via relation-aware self-attention layers to model spatial and relational context between objects.
• The model performs binary classification per object candidate to predict whether it is referred to in the current dialog turn.
• Model ensembling of five top-performing UNITER-based configurations was used to improve generalization and boost final performance on the test set.

Experimental results

Research questions

• RQ1 How effective is a UNITER-based architecture in resolving coreferences across multimodal inputs in complex, real-world dialog scenarios?
• RQ2 To what extent do rich multimodal inputs—such as object coordinates, KB entries, and scene graphs—improve coreference resolution performance?
• RQ3 Can structured relational information from scene graphs enhance the model’s understanding of spatial and contextual relationships between objects?
• RQ4 How does the proposed approach compare to sequence-based baselines like GPT-2 in terms of F1 score and robustness?

Key findings

• The proposed UNITER-based model achieved an object F1 score of 73.3% on the held-out test-std set, ranking second in the DSTC10 challenge.
• The model improved the baseline GPT-2 F1 score from 36.6% to 77.3% on the development set, demonstrating a substantial performance gain.
• Incorporating the 'previously mentioned' indicator feature alone boosted F1 from 0.674 to 0.728, indicating strong prior mention dependency.
• Adding object coordinates and KB embeddings via BERT/SBERT further improved performance, with the best single model reaching 0.674 F1 on dev-test.
• Scene graph integration yielded only marginal gains, possibly due to limited relational information in the provided graphs.
• Error analysis revealed challenges in detecting objects on walls or in cluttered scenes, especially when objects are small or occluded.