[Paper Review] Multimodal Chain-of-Thought Reasoning in Language Models
The paper introduces Multimodal-CoT, a two-stage fine-tuning framework that generates reasoning rationales from both text and vision inputs and then infers answers using those multimodal rationales, achieving state-of-the-art on ScienceQA with 1B-models.
Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.
Motivation & Objective
- Motivate multimodal (text+vision) chain-of-thought (CoT) reasoning to improve answer inference.
- Investigate why 1B-models struggle with CoT and how vision can mitigate reasoning pitfalls.
- Propose a two-stage fine-tuning framework to separate rationale generation from answer inference.
- Evaluate the approach on the ScienceQA benchmark and compare against language-only and larger models.
Proposed method
- Fine-tune a text-to-text Transformer (T5-based) in two stages: rationale generation and answer inference.
- Use a vision encoder (DETR) to extract image features and fuse them with language representations via a gated fusion mechanism.
- In the first stage, generate a rationale R from language+vision inputs; in the second stage, infer the answer conditioned on the original input and R.
- Adopt a two-stage training regime with supervised learning on annotated rationales and answers from ScienceQA.
- Incorporate vision features through attention-based interaction between language and vision representations to improve rationale quality and answer accuracy.
Experimental results
Research questions
- RQ1Can multimodal (text+vision) CoT reasoning outperform language-only CoT on a multimodal QA benchmark?
- RQ2Do 1B-models benefit from a two-stage rationale generation and answer inference framework when multimodal inputs are available?
- RQ3What is the impact of using vision features (DETR) versus captions on rationale quality and final answers?
- RQ4How does multimodal fusion (gated fusion with attention) affect reasoning and accuracy compared to text-only baselines?
Key findings
- Multimodal-CoT with vision features outperforms GPT-3.5 by 16 percentage points on ScienceQA (91.68% vs. 75.17% in the Large setting).
- Two-stage Multimodal-CoT achieves higher accuracy than one-stage baselines that directly predict answers.
- Using vision features (DETR) significantly improves rationale quality (RougeL) and final answer accuracy (84.91%), reducing hallucination-induced errors.
- Different vision features affect performance; DETR offers strong gains, while CLIP and ResNet are inferior in this setup.
- The approach generalizes across backbone models (UnifiedQA Base/Large, FLAN-T5 Base/Large) and remains effective with 1B- to ~0.7B-parameter scales.
- Ablation shows that removing the two-stage design or vision features reduces performance (e.g., w/o Two-Stage Framework drops to 82.57 Avg; w/o Vision Features drops to 70.53 Avg).
Better researchstarts right now
From reading papers to final review, dramatically reduce your research time.
No credit card · Free plan available
This review was created by AI and reviewed by human editors.