[Paper Review] IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models

Haoxuan You, Zhecan Wang+6 more|arXiv (Cornell University)|May 24, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

IdealGPT proposes an iterative, LLM-driven framework that decomposes vision-and-language reasoning into multiple rounds of sub-question generation, visual answer retrieval via a VLM, and multi-step reasoning to improve zero-shot reasoning performance. It achieves 10% higher accuracy than state-of-the-art GPT-4-like models on VCR and 15% on SNLI-VE by enabling dynamic, confidence-based iteration until reliable answers are reached.

ABSTRACT

The field of vision-and-language (VL) understanding has made unprecedented progress with end-to-end large pre-trained VL models (VLMs). However, they still fall short in zero-shot reasoning tasks that require multi-step inferencing. To achieve this goal, previous works resort to a divide-and-conquer pipeline. In this paper, we argue that previous efforts have several inherent shortcomings: 1) They rely on domain-specific sub-question decomposing models. 2) They force models to predict the final answer even if the sub-questions or sub-answers provide insufficient information. We address these limitations via IdealGPT, a framework that iteratively decomposes VL reasoning using large language models (LLMs). Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer. These three modules perform the divide-and-conquer procedure iteratively until the model is confident about the final answer to the main question. We evaluate IdealGPT on multiple challenging VL reasoning tasks under a zero-shot setting. In particular, our IdealGPT outperforms the best existing GPT-4-like models by an absolute 10% on VCR and 15% on SNLI-VE. Code is available at https://github.com/Hxyou/IdealGPT

Motivation & Objective

• To address the limitations of end-to-end vision-and-language models in zero-shot, multi-step reasoning tasks.
• To overcome the reliance on task-specific sub-question models and the assumption of sufficient evidence in single-round reasoning.
• To enhance interpretability, modularity, and robustness in vision-and-language reasoning through iterative, confidence-driven refinement.
• To reduce hallucinations and spurious predictions by deferring final answers until sufficient visual and linguistic evidence is gathered.

Proposed method

• IdealGPT employs three LLM agents: a Questioner to generate sub-questions, an Answerer (a pre-trained VLM) to provide visual answers, and a Reasoner to assess evidence and guide further iterations.
• The framework operates iteratively: if the Reasoner lacks confidence, it triggers a new round by prompting the Questioner to generate more targeted sub-questions.
• Each iteration collects new sub-answers, which the Reasoner evaluates for coherence and sufficiency before deciding to continue or conclude.
• The process continues until the Reasoner is confident in the final answer or a maximum iteration threshold is reached.
• The system uses GPT-4 as the Questioner and Reasoner, and BLIP-2 as the Answerer, with prompts manually designed for consistency and clarity.
• The method is modular, allowing replacement of components with stronger LLMs or VLMs to improve performance.

Experimental results

Research questions

• RQ1 Can iterative, LLM-driven decomposition improve zero-shot vision-and-language reasoning beyond single-step end-to-end models?
• RQ2 Does confidence-based iteration reduce hallucinations and improve reasoning robustness in vision-and-language tasks?
• RQ3 How does the choice of VLM as the Answerer affect performance in the iterative framework?
• RQ4 To what extent does the quality of generated image captions influence sub-question generation and final reasoning?
• RQ5 Can a modular, multi-agent framework with dynamic feedback loops outperform fixed-pipeline or end-to-end VLMs in complex reasoning tasks?

Key findings

• IdealGPT outperforms the best existing GPT-4-like models by an absolute 10% on the VCR benchmark under zero-shot settings.
• IdealGPT achieves a 15% absolute improvement over state-of-the-art models on the SNLI-VE benchmark in zero-shot evaluation.
• The iterative decomposition mechanism improves accuracy by up to 6% compared to a single-pass (max_iterations=1) setup, with 1.8 average iterations per sample.
• BLIP-2 outperforms MiniGPT4 and LLaVA as the Answerer despite their stronger captioning abilities, due to lower hallucination rates in sub-answer generation.
• MiniGPT4 and LLaVA generate longer, more informative, but less accurate captions, which mislead the Questioner and Reasoner, reducing overall performance.
• The ablation study confirms that the quality of image captions significantly impacts downstream reasoning, with shorter, more conservative captions yielding better results.