[Paper Review] REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory

Ziniu Hu, Ahmet İşcen+7 more|arXiv (Cornell University)|Dec 10, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

ReVeaL proposes an end-to-end retrieval-augmented visual-language model that pre-trains a unified multimodal memory from diverse sources—such as image-text pairs, knowledge graphs, and Wikipedia passages—enabling efficient retrieval and generation. By jointly training the retriever and generator with an attentive fusion mechanism, ReVeaL achieves state-of-the-art performance on OKVQA (59.1% accuracy) with significantly fewer parameters than prior methods.

ABSTRACT

In this paper, we propose an end-to-end Retrieval-Augmented Visual Language Model (REVEAL) that learns to encode world knowledge into a large-scale memory, and to retrieve from it to answer knowledge-intensive queries. REVEAL consists of four key components: the memory, the encoder, the retriever and the generator. The large-scale memory encodes various sources of multimodal world knowledge (e.g. image-text pairs, question answering pairs, knowledge graph triplets, etc) via a unified encoder. The retriever finds the most relevant knowledge entries in the memory, and the generator fuses the retrieved knowledge with the input query to produce the output. A key novelty in our approach is that the memory, encoder, retriever and generator are all pre-trained end-to-end on a massive amount of data. Furthermore, our approach can use a diverse set of multimodal knowledge sources, which is shown to result in significant gains. We show that REVEAL achieves state-of-the-art results on visual question answering and image captioning.

Motivation & Objective

• To address the limitations of large-scale visual-language models that require massive retraining when world knowledge is updated.
• To enable efficient, scalable, and dynamic knowledge retrieval by decoupling knowledge storage from model parameters.
• To improve performance on knowledge-intensive visual-language tasks by leveraging diverse multimodal knowledge sources.
• To develop a differentiable, end-to-end training paradigm for retrieval-augmented visual-language models with joint optimization of retriever and generator.
• To enable rapid knowledge updates without retraining model parameters by modifying only the external memory.

Proposed method

• The model uses a unified encoder to embed diverse multimodal knowledge sources—image-text pairs, Wikipedia passages, knowledge graph triplets—into a shared memory space.
• A retriever module uses multimodal query embeddings to retrieve the most relevant knowledge entries from the memory using a differentiable attention-based fusion mechanism.
• An attentive fusion layer integrates retrieval scores directly into the self-attention mechanism, enabling end-to-end backpropagation through the retriever.
• The generator fuses the retrieved knowledge with the input query to produce answers or captions, trained via a language modeling objective.
• The entire system—memory, encoder, retriever, and generator—is pre-trained jointly on a massive-scale image-text corpus with multiple knowledge sources.
• Knowledge updates are supported by modifying the memory without retraining the model parameters, enabling dynamic adaptation.

Figure 2 : The overall workflow of R e V ea L consists of four main steps: (a) encode a multimodal input into a sequence of token embeddings and a summarized query embedding; (b) encode each knowledge entry from different corpus into unified key and value embedding pairs, where key is used to index

Experimental results

Research questions

• RQ1 Can a retrieval-augmented visual-language model achieve state-of-the-art performance on knowledge-intensive visual question answering and image captioning tasks?
• RQ2 Does end-to-end pre-training of the retriever with the generator lead to better retrieval and generation performance than frozen or distillation-based methods?
• RQ3 Can a unified multimodal memory effectively integrate diverse knowledge sources (e.g., text, images, knowledge graphs) for improved reasoning?
• RQ4 How efficiently can the model adapt to new or updated knowledge without retraining its parameters?
• RQ5 Does the attentive fusion mechanism improve retrieval quality and downstream task accuracy compared to independent retrieval scoring?

Key findings

• ReVeaL achieves a new state-of-the-art accuracy of 59.1% on the OKVQA benchmark, significantly outperforming prior methods.
• The model achieves this result with an order of magnitude fewer parameters than previous SOTA approaches, demonstrating high parameter efficiency.
• The attentive fusion method achieves 72.6% retrieval accuracy at top-10 and 89.4% at top-100 on the WIT dataset, outperforming baseline methods like Attention Distill and Perplexity Distill.
• When knowledge is removed from the memory, model accuracy drops to 45.1%; upon reintroduction, accuracy recovers to 51.8%, proving the model’s ability to adapt to knowledge updates without retraining.
• The proposed method has a computational cost (120 GFLOPs) comparable to Attention Distill but significantly lower than EMDR2 and Perplexity Distill, indicating high efficiency.
• The model demonstrates strong generalization and robustness, with performance gains from using diverse multimodal knowledge sources in the memory.

Figure 3 : Detailed procedure of attentive knowledge fusion module. We inject retrieval probability as a prior to knowledge token embeddings, so the retriever can receive gradients via back-propagating over {self/cross}-attention part.