[Paper Review] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Pan Lu, Swaroop Mishra+7 more|arXiv (Cornell University)|Sep 20, 2022

Topic ModelingComputer Science214 citations

TL;DR

This paper introduces ScienceQA, a large multimodal science QA dataset with lectures and explanations, and shows that chain-of-thought generation in language models improves performance in few-shot and fine-tuning settings, with GPT-3 and UnifiedQA benefiting from CoT explanations.

ABSTRACT

When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions. ScienceQA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data. The data and code are available at https://scienceqa.github.io.

Motivation & Objective

• Create a large-scale, multimodal science QA dataset with annotated lectures and explanations to reveal reasoning paths.
• Evaluate how chain-of-thought (CoT) generation in language models affects question answering accuracy on ScienceQA.
• Demonstrate the potential of explanations to improve learning efficiency and data efficiency in few-shot and fine-tuning regimes.

Proposed method

• Assemble ScienceQA with 21,208 multimodal multiple-choice questions spanning natural, social, and language sciences, each tied to lectures and explanations.
• Benchmark a range of baselines including VQA models and large language models (UnifiedQA and GPT-3) on QCM-style prompts.
• Modify UnifiedQA to generate an answer plus a lecture and explanation (CoT) during training and evaluation.
• Apply GPT-3 prompting with chain-of-thought to generate answer, lecture, and explanation, comparing to standard prompts.
• Analyze generated explanations with automatic metrics (BLEU-1/4, ROUGE-L, Similarity) and human evaluation on relevance, correctness, and completeness.
• Explore upper-bound scenarios by feeding gold explanations into prompts to measure potential gains.

Experimental results

Research questions

• RQ1 Can a large-scale multimodal science QA dataset with grounded lectures and explanations support multi-hop reasoning assessment and model interpretability?
• RQ2 Do chain-of-thought explanations improve QA accuracy for multimodal science questions in few-shot and fine-tuning regimes?
• RQ3 To what extent can explanations be leveraged to enhance learning efficiency and data efficiency for language models on ScienceQA?
• RQ4 What is the gap between machine and human performance on multimodal science questions with explanations?

Key findings

• ScienceQA contains 21,208 multimodal questions across natural, social, and language sciences with rich context (text and/or images) and annotated lectures and explanations.
• UnifiedQA with a chain-of-thought (CoT) improves average accuracy by 3.99% when fine-tuned (QCM → ALE) versus without CoT.
• GPT-3 with CoT prompting achieves 75.17% average accuracy on ScienceQA in a 2-shot setting, surpassing non-CoT prompts.
• Including ground-truth explanations in prompts yields up to 18.96% absolute improvement in GPT-3 few-shot performance (upper bound analysis).
• Explanations help models learn from less data: UnifiedQA with CoT reaches similar accuracy with 40% of the training data.
• 65.2% of GPT-3 (CoT) generated explanations meet gold standards in human evaluation (relevant, correct, complete).
• Humans significantly outperform all models, with about an 20-point gap on image-context questions.