[Paper Review] Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation

Qian Yang, Qian Chen+3 more|arXiv (Cornell University)|Dec 16, 2022

Topic Modeling4 citations

TL;DR

This paper proposes SKURG, a unified framework for multi-modal multi-hop question answering that enhances retrieval and generation through entity-centered knowledge fusion and a joint retrieval-generation decoder. By aligning multi-modal inputs via a dynamically constructed knowledge graph and enabling adaptive, interdependent retrieval and answer generation, SKURG achieves state-of-the-art performance on MultimodalQA and WebQA with fewer parameters.

ABSTRACT

Multi-modal multi-hop question answering involves answering a question by reasoning over multiple input sources from different modalities. Existing methods often retrieve evidences separately and then use a language model to generate an answer based on the retrieved evidences, and thus do not adequately connect candidates and are unable to model the interdependent relations during retrieval. Moreover, the pipelined approaches of retrieval and generation might result in poor generation performance when retrieval performance is low. To address these issues, we propose a Structured Knowledge and Unified Retrieval-Generation (SKURG) approach. SKURG employs an Entity-centered Fusion Encoder to align sources from different modalities using shared entities. It then uses a unified Retrieval-Generation Decoder to integrate intermediate retrieval results for answer generation and also adaptively determine the number of retrieval steps. Extensive experiments on two representative multi-modal multi-hop QA datasets MultimodalQA and WebQA demonstrate that SKURG outperforms the state-of-the-art models in both source retrieval and answer generation performance with fewer parameters. Our code is available at https://github.com/HITsz-TMG/SKURG.

Motivation & Objective

• To address the limitations of existing multi-modal multi-hop QA systems in modeling cross-modal relations during retrieval.
• To overcome error propagation and rigid retrieval step counts in pipelined retrieval-generation architectures.
• To unify evidence retrieval and answer generation into a single, adaptive process that leverages shared entity representations.
• To improve reasoning over multi-modal sources by constructing a knowledge graph directly from input data rather than relying on external knowledge.
• To enhance performance on complex, multi-hop questions requiring integration of visual and textual evidence.

Proposed method

• An Entity-centered Fusion Encoder (EF-Enc) constructs a knowledge graph from input sources using named entity recognition and relation extraction, aligning entities across modalities.
• EF-Enc fuses multi-modal inputs into a shared semantic space by linking entities to their source modalities (e.g., image, text).
• A Unified Retrieval-Generation Decoder (RG-Dec) uses a pointer mechanism to retrieve relevant evidences and generate answers in an end-to-end, interdependent manner.
• RG-Dec adaptively determines the number of retrieval steps by attending to both source representations and fused entity embeddings.
• The model jointly optimizes retrieval and generation, allowing answer generation to guide evidence retrieval through shared attention.
• The framework uses a sequence-to-sequence transformer-based architecture with shared parameters for retrieval and generation, enabling parameter efficiency.

Experimental results

Research questions

• RQ1 Can a unified retrieval-generation architecture improve multi-modal multi-hop QA by reducing error propagation from separate retrieval and generation stages?
• RQ2 How does entity-centered knowledge fusion enhance cross-modal alignment and reasoning in multi-hop QA?
• RQ3 To what extent does adaptive, dynamic retrieval improve performance compared to fixed-hop retrieval in multi-modal settings?
• RQ4 Does end-to-end joint optimization of retrieval and generation lead to better answer generation and evidence retrieval than pipelined approaches?
• RQ5 How does the model perform on short-answer vs. long-answer generation tasks in multi-modal QA?

Key findings

• SKURG outperforms state-of-the-art models on both MultimodalQA and WebQA in both retrieval F1 and answer generation performance.
• On MultimodalQA, SKURG achieves a 9.3 absolute improvement in retrieval F1 and a 3.5 absolute gain in answer F1 over the baseline.
• On WebQA, SKURG achieves a 11.1 absolute improvement in retrieval F1 for long answers (>20 tokens), indicating stronger retrieval capability for complex queries.
• The ablation study shows that using gold entity alignments instead of predicted ones improves performance, confirming the importance of accurate entity grounding.
• SKURG shows larger gains on MultimodalQA than on WebQA, likely due to the former's greater reliance on cross-modal reasoning and shorter answer spans.
• Case studies demonstrate that EF-Enc enables correct image-source alignment, while RG-Dec enables retrieval of semantically linked but non-identical evidences (e.g., brand ambassador from image and text).