[Paper Review] MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering

Janak Kapuriya, Chhavi Kirtani+8 more|arXiv (Cornell University)|Apr 19, 2024

Topic Modeling4 citations

TL;DR

This paper proposes MM-PhyRLHF, a reinforcement learning framework integrating image captioning and Reinforcement Learning from Human Feedback (RLHF) to improve multimodal physics question-answering in LLaVA-based models. By enhancing visual understanding through detailed image captions and refining reasoning via human preference-based reward modeling, the approach boosts accuracy to 83.28% on the MM-PhyQA dataset, significantly outperforming supervised fine-tuning alone.

ABSTRACT

Recent advancements in LLMs have shown their significant potential in tasks like text summarization and generation. Yet, they often encounter difficulty while solving complex physics problems that require arithmetic calculation and a good understanding of concepts. Moreover, many physics problems include images that contain important details required to understand the problem's context. We propose an LMM-based chatbot to answer multimodal physics MCQs. For domain adaptation, we utilize the MM-PhyQA dataset comprising Indian high school-level multimodal physics problems. To improve the LMM's performance, we experiment with two techniques, RLHF (Reinforcement Learning from Human Feedback) and Image Captioning. In image captioning, we add a detailed explanation of the diagram in each image, minimizing hallucinations and image processing errors. We further explore the integration of Reinforcement Learning from Human Feedback (RLHF) methodology inspired by the ranking approach in RLHF to enhance the human-like problem-solving abilities of the models. The RLHF approach incorporates human feedback into the learning process of LLMs, improving the model's problem-solving skills, truthfulness, and reasoning capabilities, minimizing the hallucinations in the answers, and improving the quality instead of using vanilla-supervised fine-tuned models. We employ the LLaVA open-source model to answer multimodal physics MCQs and compare the performance with and without using RLHF.

Motivation & Objective

• To address the limitations of LLMs in solving complex, multimodal physics problems requiring arithmetic reasoning and visual understanding.
• To reduce hallucinations and improve factual consistency in multimodal physics question-answering by enriching images with detailed captions.
• To enhance reasoning and response quality through Reinforcement Learning from Human Feedback (RLHF), aligning model outputs with human preferences.
• To evaluate the combined impact of image captioning and RLHF on performance in Indian high school-level physics MCQs.
• To develop a robust, open-source educational AI system tailored for physics education using multimodal LLMs.

Proposed method

• Fine-tuned LLaVA-1.5 models using a modified MM-PhyQA dataset enriched with detailed image captions generated via the Infi-MM captioning model.
• Applied Supervised Fine-Tuning (SFT) with three input combinations: text + image, text + caption, and text + image + caption.
• Integrated RLHF using a preference dataset derived from human rankings of response quality, focusing on reasoning coherence and prompt adherence.
• Trained a Reward Model (RM) to predict response quality based on human preference rankings, which then guided policy optimization via Proximal Policy Optimization (PPO).
• Employed LoRA (Low-Rank Adaptation) for efficient fine-tuning, testing multiple LoRA ranks and model sizes (7B, 13B) to balance speed and performance.
• Combined multi-image Chain-of-Thought prompting with RLHF and captioning to improve step-by-step reasoning in physics problem-solving.

Figure 1. Preference dataset Pipeline

Experimental results

Research questions

• RQ1 How does adding detailed image captions to multimodal physics datasets affect LLM performance in reasoning and accuracy?
• RQ2 To what extent does integrating RLHF improve the factual consistency, reasoning quality, and truthfulness of LLM-generated answers in physics QA?
• RQ3 What is the relative contribution of image captioning versus RLHF in enhancing multimodal physics question-answering accuracy?
• RQ4 How do different LLaVA model sizes and LoRA configurations impact the effectiveness of the proposed framework?
• RQ5 Can the integration of RLHF and captioning reduce hallucinations in multimodal physics problem-solving compared to standard supervised fine-tuning?

Key findings

• The highest accuracy of 83.28% was achieved using the LLaVA-1.5 13b model fine-tuned on text, image, and caption inputs with RLHF, significantly outperforming baseline SFT-only approaches.
• Supervised fine-tuning with image captions alone (no image input) achieved 74.56% accuracy with the 13b LoRA large model, indicating that textual visual descriptions can compensate for missing visual data.
• The addition of image captions to the training data increased SFT accuracy from 53.3% (text + image only) to 82.52% (text + image + caption), demonstrating a 29.22 percentage point improvement.
• RLHF integration further boosted performance, with the best-performing setup (text + image + caption + RLHF) achieving 83.28% accuracy, indicating that human feedback refinement enhances reasoning and factual correctness.
• The 13b model with LoRA large (rank 128) outperformed smaller LoRA configurations, suggesting that higher parameter efficiency in adaptation improves performance in complex reasoning tasks.
• The combination of image captioning and RLHF reduced hallucinations and improved response coherence, as validated by human preference-based reward modeling.