[Paper Review] Baichuan-M1: Pushing the Medical Capability of Large Language Models

Bingning Wang, Haizhou Zhao+41 more|ArXiv.org|Feb 18, 2025

Radiomics and Machine Learning in Medical Imaging5 citations

TL;DR

Baichuan-M1 is a medical-specialized LLM trained from scratch on 20 trillion tokens (including 1T medical data) to boost medical capabilities while maintaining general tasks, with Baichuan-M1-14B released for public use.

ABSTRACT

The current generation of large language models (LLMs) is typically designed for broad, general-purpose applications, while domain-specific LLMs, especially in vertical fields like medicine, remain relatively scarce. In particular, the development of highly efficient and practical LLMs for the medical domain is challenging due to the complexity of medical knowledge and the limited availability of high-quality data. To bridge this gap, we introduce Baichuan-M1, a series of large language models specifically optimized for medical applications. Unlike traditional approaches that simply continue pretraining on existing models or apply post-training to a general base model, Baichuan-M1 is trained from scratch with a dedicated focus on enhancing medical capabilities. Our model is trained on 20 trillion tokens and incorporates a range of effective training methods that strike a balance between general capabilities and medical expertise. As a result, Baichuan-M1 not only performs strongly across general domains such as mathematics and coding but also excels in specialized medical fields. We have open-sourced Baichuan-M1-14B, a mini version of our model, which can be accessed through the following links.

Motivation & Objective

• Motivate the need for domain-specific LLMs in medicine due to data and knowledge complexity.
• Describe a from-scratch training approach focused on medical capabilities while preserving general skills.
• Show data collection, filtering, synthetic data strategies, and curriculum-style training to improve medical reasoning.

Proposed method

• Train Baichuan-M1 from scratch with a dedicated focus on medical knowledge and general capabilities.
• Use a three-stage pre-training curriculum with increasing medical data and longer context windows.
• Apply advanced tokenizer design combining general and medical vocabularies (vocabulary size 133,120).
• Employ a mixed attention architecture alternating global and sliding window attention to balance efficiency and performance.
• Incorporate synthetic data generation pipelines (knowledge extraction, QA, long CoT) and real-world case reasoning data to enhance medical reasoning.
• Perform multi-stage supervised fine-tuning for general and medical instruction following safety and alignment goals.

Experimental results

Research questions

• RQ1 How can a large language model be built from scratch to maximize medical capability without sacrificing general performance?
• RQ2 What data strategies (quality, upsampling, synthetic data) best improve medical knowledge, reasoning, and long-context handling?
• RQ3 What architectural choices (hybrid attention, larger KV cache, RoPE base) yield better in-context learning and inference efficiency for medical tasks?

Key findings

• Baichuan-M1-14B demonstrates strong performance on general tasks (math, coding) and excels in medical domains.
• A deduplication plus upsampling data strategy improves performance over deduplication alone.
• 3-stage training with increasing medical data and context length supports long-context medical reasoning and accuracy.
• Synthetic medical data with long chain-of-thought generation improves reasoning capabilities and aligns with medical knowledge.
• A hybrid global and sliding-window attention setup maintains long-context retrieval while improving short-context benchmarks.
• Public release of Baichuan-M1-14B-Base, Baichuan-M1-14B-Base (Instruct) for community use.