[Paper Review] BioXP-0.5B: Explainable Medical-AI via RL-GRPO

Zhihong Shao, Peiyi Wang+9 more|arXiv (Cornell University)|Feb 5, 2024

Mathematics, Computing, and Information Processing66 citations

TL;DR

Introduces BioXP-0.5B, an explainable Medical-AI model built with RL-GRPO to enhance reasoning and memory efficiency, achieving strong medical math and reasoning benchmarks.

ABSTRACT

BioXP-0.5B is a 🤗 Medical-AI model trained using our two-stage fine-tuning approach: Supervised Fine-Tuning (SFT): The model was initially fine-tuned on labeled data(MedMCQA) to achieve strong baseline accuracy on multiple-choice medical QA tasks. Group Relative Policy Optimization (GRPO): In the second stage, GRPO was applied to further align the model with human-like reasoning patterns. This reinforcement learning technique enhances the model’s ability to generate coherent, high-quality explanations and improve answer reliability.

Motivation & Objective

• Build a domain-specific language model for medical reasoning with interpretable outputs.
• Demonstrate scalable math pre-training using a large, curated corpus.
• Develop and evaluate an efficient RL method (GRPO) to improve mathematical reasoning with reduced memory overhead.
• Show that math-centric pre-training boosts both mathematical and general reasoning benchmarks.

Proposed method

• Create a math-focused pre-training corpus (DeepSeekMath Corpus) from Common Crawl using a fastText classifier trained on OpenWebMath as seed.
• Pre-train a base model DeepSeekMath-Base 7B initialized from DeepSeek-Coder-Base-v1.5 7B for 500B tokens.
• Apply mathematical instruction tuning (CoT, PoT, tool-integrated reasoning) to obtain DeepSeekMath-Instruct 7B.
• Develop Group Relative Policy Optimization (GRPO), a PPO variant that uses group scores as baseline and omits a separate value function.
• Evaluate English and Chinese benchmarks across math solving with and without tools, formal proving, and general NLU/Code tasks.
• Provide a unified framework to compare RFT, DPO, PPO, and GRPO and analyze RL elements.

Experimental results

Research questions

• RQ1 How does large-scale math pre-training on a multilingual corpus affect mathematical reasoning on English and Chinese benchmarks?
• RQ2 Can GRPO improve Math RL efficiency and performance without a separate critic model?
• RQ3 Does math-focused pre-training enhance general reasoning and coding capabilities beyond mathematics?
• RQ4 What is the impact of instruction tuning with CoT/PoT/tool-based reasoning on open-source models?
• RQ5 How do different RL supervision strategies (outcome vs process) influence performance in math tasks?

Key findings

• DeepSeekMath-Base 7B achieves strong English and Chinese math performance, surpassing several open-source baselines on multiple benchmarks.
• DeepSeekMath-Instruct 7B and DeepSeekMath-RL 7B achieve top open-source performance, with RL enabling substantial gains on MATH and other tasks.
• GRPO, a memory-efficient PPO variant, yields significant improvements using only English instruction-tuning data, boosting in-domain and out-of-domain math performance.
• Math pre-training on the DeepSeekMath Corpus improves both math-specific and general reasoning abilities, including MMLU and BBH benchmarks.
• Tool-enabled reasoning and chain-of-thought prompts further enhance math problem solving, with DeepSeekMath-RL outperforming many larger or similarly sized models.
• The authors provide a unified paradigm showing how RFT, DPO, PPO, and GRPO relate as direct or simplified RL methods, and discuss reasons for RL effectiveness.