[Paper Review] Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task

Stan Weixian Lei, Difei Gao+5 more|arXiv (Cornell University)|Aug 24, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes CLOVE, a new benchmark for continual learning in Visual Question Answering (VQA) that supports scene- and function-incremental settings. It introduces Scene Graph as Prompt (SGP), a symbolic replay method using scene graphs as prompts to replay knowledge without real images, enabling effective continual learning by preserving reasoning mechanisms and preventing catastrophic forgetting in multi-modal VQA models.

ABSTRACT

VQA is an ambitious task aiming to answer any image-related question. However, in reality, it is hard to build such a system once for all since the needs of users are continuously updated, and the system has to implement new functions. Thus, Continual Learning (CL) ability is a must in developing advanced VQA systems. Recently, a pioneer work split a VQA dataset into disjoint answer sets to study this topic. However, CL on VQA involves not only the expansion of label sets (new Answer sets). It is crucial to study how to answer questions when deploying VQA systems to new environments (new Visual scenes) and how to answer questions requiring new functions (new Question types). Thus, we propose CLOVE, a benchmark for Continual Learning On Visual quEstion answering, which contains scene- and function-incremental settings for the two aforementioned CL scenarios. In terms of methodology, the main difference between CL on VQA and classification is that the former additionally involves expanding and preventing forgetting of reasoning mechanisms, while the latter focusing on class representation. Thus, we propose a real-data-free replay-based method tailored for CL on VQA, named Scene Graph as Prompt for Symbolic Replay. Using a piece of scene graph as a prompt, it replays pseudo scene graphs to represent the past images, along with correlated QA pairs. A unified VQA model is also proposed to utilize the current and replayed data to enhance its QA ability. Finally, experimental results reveal challenges in CLOVE and demonstrate the effectiveness of our method. The dataset and code will be available at https://github.com/showlab/CLVQA.

Motivation & Objective

• Address the lack of benchmarks for continual learning in VQA that support both scene and function incremental settings.
• Overcome the challenge of catastrophic forgetting in VQA models when learning new visual scenes or reasoning functions.
• Develop a real-data-free replay mechanism that preserves reasoning capabilities without relying on image generation.
• Enable VQA models to adapt to new environments and new question types over time, mimicking real-world deployment scenarios.

Proposed method

• Propose CLOVE, a new benchmark with two settings: scene-incremental (6 scenes) and function-incremental (6 functions) for continual VQA.
• Design a symbolic replay mechanism using scene graphs as prompts to represent past knowledge, replacing real image replay.
• Develop a Symbolic Replay Model (SRM) based on a language model to learn task-specific mappings between scene graphs and QA pairs.
• Use a unified VQA model (UniVQA) that processes both current and replayed scene graph–QA inputs to enhance continual learning.
• Implement prompt-based inference where a randomly sampled scene graph relationship triggers replay of past knowledge.
• Leverage structured, symbolic representations to avoid the limitations of low-quality image generation in pseudo-replay methods.

Experimental results

Research questions

• RQ1 How can continual learning in VQA be effectively evaluated under realistic scene-incremental and function-incremental settings?
• RQ2 Can symbolic replay using scene graphs outperform traditional image-based replay in preventing forgetting during continual VQA training?
• RQ3 To what extent can a unified VQA model generalize across new visual scenes and new reasoning functions using symbolic prompts?
• RQ4 What are the key challenges in continual VQA when expanding both visual environments and functional capabilities simultaneously?
• RQ5 How does the proposed SGP method preserve reasoning mechanisms while avoiding catastrophic forgetting in multi-modal, multi-task VQA?

Key findings

• The CLOVE benchmark reveals significant challenges in continual VQA, particularly in maintaining performance across both scene and function increments.
• The proposed SGP method achieves superior performance compared to baseline replay and regularization methods on both CLOVE-scene and CLOVE-function settings.
• Symbolic replay using scene graphs effectively preserves reasoning mechanisms and reduces catastrophic forgetting, even without real image data.
• The unified VQA model (UniVQA) successfully integrates current and replayed knowledge, demonstrating robust generalization across diverse visual and functional tasks.
• Answer distributions in CLOVE show high class imbalance, highlighting the difficulty of continual learning in open-ended VQA tasks.
• Qualitative analysis confirms that SGP generates plausible and diverse QA pairs from scene graphs, supporting effective knowledge replay.