[Paper Review] Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction

Qian Li, Shu Guo+4 more|arXiv (Cornell University)|Jun 19, 2023

Natural Language Processing TechniquesComputer Science3 citations

TL;DR

This paper proposes DGF-PT, a novel multi-modal relation extraction framework that uses dual-gated fusion and prefix-tuning to capture fine-grained correlations among entity pairs, text, and visual inputs. By employing entity- and object-oriented prefixes and a dual-gated fusion module, the model effectively identifies and integrates helpful visual cues while filtering out noisy or misleading information, achieving state-of-the-art performance, especially in few-shot settings.

ABSTRACT

Multi-Modal Relation Extraction (MMRE) aims at identifying the relation between two entities in texts that contain visual clues. Rich visual content is valuable for the MMRE task, but existing works cannot well model finer associations among different modalities, failing to capture the truly helpful visual information and thus limiting relation extraction performance. In this paper, we propose a novel MMRE framework to better capture the deeper correlations of text, entity pair, and image/objects, so as to mine more helpful information for the task, termed as DGF-PT. We first propose a prompt-based autoregressive encoder, which builds the associations of intra-modal and inter-modal features related to the task, respectively by entity-oriented and object-oriented prefixes. To better integrate helpful visual information, we design a dual-gated fusion module to distinguish the importance of image/objects and further enrich text representations. In addition, a generative decoder is introduced with entity type restriction on relations, better filtering out candidates. Extensive experiments conducted on the benchmark dataset show that our approach achieves excellent performance compared to strong competitors, even in the few-shot situation.

Motivation & Objective

• To address the challenge of noisy and unhelpful visual inputs in multi-modal relation extraction (MMRE), which can degrade model performance.
• To model finer-grained associations between entity pairs, text, and visual objects to improve relation prediction.
• To develop a method that effectively distinguishes helpful visual content from irrelevant or misleading visual signals.
• To enhance performance in low-resource (few-shot) scenarios by leveraging prompt-based learning and structured fusion.
• To restrict relation candidate generation using entity type constraints for improved filtering and accuracy.

Proposed method

• Introduces a prompt-based autoregressive encoder with two distinct prefix types: entity-oriented and object-oriented prefixes to model intra-modal and inter-modal associations.
• Employs a dual-gated fusion module with a local object-importance gate and a global image-relevance gate to dynamically weigh and integrate visual features based on their relevance.
• Designs a generative decoder that restricts relation candidates using entity type information to improve prediction precision.
• Applies a joint objective to align the representation distributions before and after fusion, enhancing feature consistency and model robustness.
• Utilizes prefix-tuning to adapt large language models efficiently without full fine-tuning, supporting few-shot generalization.
• Leverages visual relation information from object detection and image features to enrich text representations in a task-aware manner.

Experimental results

Research questions

• RQ1 How can deeper correlations among entity pairs, text, and visual objects be effectively modeled in multi-modal relation extraction?
• RQ2 Can a dual-gated fusion mechanism improve the selection of helpful visual features while suppressing unhelpful or misleading ones?
• RQ3 To what extent does prefix-tuning enhance performance in low-resource (few-shot) relation extraction scenarios?
• RQ4 How does entity-type-aware decoding improve relation candidate filtering and prediction accuracy?
• RQ5 Can the integration of intra-modal and inter-modal associations via prefix-tuning lead to better generalization and robustness in MMRE?

Key findings

• DGF-PT achieves state-of-the-art performance on the benchmark MNRE dataset, outperforming strong baselines in both full-shot and few-shot settings.
• The model demonstrates superior generalization in low-resource scenarios, with consistent performance gains as sample size decreases.
• The dual-gated fusion mechanism effectively filters out unhelpful visual content, reducing the negative impact of noisy images on predictions.
• Case studies confirm that DGF-PT correctly identifies relevant visual cues for different entity pairs, even when the same image is used with varying entity pairs.
• The use of entity-type-restricted decoding significantly improves candidate filtering, reducing incorrect predictions on ambiguous relations.
• The model’s performance improves with increased visual data, indicating effective utilization of visual features compared to prefix-based baselines.