[Paper Review] Baichuan 2: Open Large-scale Language Models

A. Y. Yang, Bin Xiao+53 more|arXiv (Cornell University)|Sep 19, 2023

Topic Modeling125 citations

TL;DR

Baichuan 2 presents open, multilingual LLMs with 7B and 13B parameters trained on 2.6T tokens, achieving competitive or superior results to open models and strong domain performance in medicine and law; includes released checkpoints and chat variants aligned with human preferences.

ABSTRACT

Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2.

Motivation & Objective

• Address the need for open multilingual LLMs beyond English-dominated models.
• Scale training data and model size to improve general and domain-specific performance.
• Develop architecture and training optimizations to enable efficient large-scale pre-training and alignment.
• Release model checkpoints and chat variants to advance safety, reproducibility, and research collaboration.

Proposed method

• Propose Baichuan 2 with two sizes (7B and 13B) trained from scratch on 2.6T tokens across multilingual data.
• Modify Transformer architecture with SwiGLU activations, LayerNorm/RMSNorm, memory-efficient attention, and enhanced tokenizer (125,696 vocab size).
• Use RoPE for Baichuan 2-7B and ALiBi for Baichuan 2-13B, with optimized attention via xFormers.
• Apply NormHead and Max-z loss to stabilize training and ensure robust inference.
• Employ distributed training with tensor parallelism and ZeRO-based data parallelism, plus memory-splitting techniques and mixed precision (BF16/Float32) for efficiency.
• Implement alignment via supervised fine-tuning (SFT) followed by RLHF using PPO, a reward model with multi-category prompts, and a 350-iteration policy optimization for chat models.

Experimental results

Research questions

• RQ1 How does Baichuan 2 perform on general benchmarks relative to other open-sized LLMs?
• RQ2 What is the impact of large-scale pre-training data on multilingual and domain-specific capabilities?
• RQ3 Do architectural and training optimizations yield measurable gains in efficiency and stability for 7B and 13B models?
• RQ4 How effective is the alignment pipeline (SFT + RLHF) in producing safe and helpful chat models?
• RQ5 What is the comparative performance of Baichuan 2 in vertical domains like medicine and law?

Key findings

• Baichuan 2-7B-Base and Baichuan 2-13B-Base outperform other similarly sized open models on multiple benchmarks (e.g., MMLU, CMMLU, GSM8K, HumanEval).
• Baichuan 2-7B-Base reaches strong scores in law and medicine, often surpassing non-GPT-4 baselines and nearing GPT-4 in some Chinese-language tasks.
• Baichuan 2 demonstrates significant gains over Baichuan 1 across general and domain benchmarks, including near-doubling results on GSM8K and HumanEval.
• Multilingual evaluation on Flores-101 shows Baichuan 2-7B-Base surpasses peers on all seven tasks; Baichuan 2-13B-Base exceeds peers in several tasks, with Chinese-English capabilities approaching GPT-4 in some pairs.
• Code and math abilities improve substantially with Baichuan 2, with 7B/13B bases outperforming many contemporaries in respective domains.
• The project provides open model checkpoints from 200B to 2.6T tokens to illuminate training dynamics and support further research.