[Paper Review] InternLM2 Technical Report

Zheng Cai, Maosong Cao+98 more|arXiv (Cornell University)|Mar 26, 2024

Artificial Intelligence in Healthcare and Education27 citations

TL;DR

InternLM2 is an open-source LLM that surpasses prior open models across 30 benchmarks, supports long contexts up to 200k, and uses COOL RLHF for alignment.

ABSTRACT

The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advancements in open-source models has been challenging. This paper introduces InternLM2, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types including text, code, and long-context data. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages, exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test. InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF) strategy that addresses conflicting human preferences and reward hacking. By releasing InternLM2 models in different training stages and model sizes, we provide the community with insights into the model's evolution.

Motivation & Objective

• Demonstrate open-source LLM performance across multiple benchmarks and dimensions.
• Detail diverse pre-training data preparation including text, code, and long-context data.
• Describe long-context training techniques enabling 32k+ context windows and 200k needle-in-a-haystack capability.
• Present alignment methods, including SFT and COOL RLHF, to handle conflicting human preferences.
• Share training infrastructure (InternEvo) and model evolution across stages to guide community adoption.

Proposed method

• Describe data processing pipeline with formatting, deduplication, safety and quality filtering for text data.
• Explain pre-training tokenization based on a GPT-4 style cl100k vocabulary with Chinese augmentation.
• Outline long-context pre-training via 4k then 32k context data, and Locality-Sensitive Hashing for deduplication.
• Introduce Grouped-Query Attention (GQA) for long-context inference and tensor-parallel layout choices.
• Detail Supervised Fine-Tuning (SFT) and Conditional Online RLHF (COOL RLHF) with a conditional reward model and PPO updates.
• Present the InternEvo training framework with memory efficiency, fault tolerance, and interactive RLHF setup.

Experimental results

Research questions

• RQ1 How does InternLM2 perform across 6 dimensions and 30 benchmarks compared to open-source rivals?
• RQ2 What data processing and filtering strategies best support safe, high-quality pre-training data?
• RQ3 Can long-context models (up to 200k tokens) be effectively trained and leveraged in practice?
• RQ4 How does COOL RLHF address conflicting human preferences and reward hacking in alignment?
• RQ5 What infrastructure and model evolution practices enhance scalability and reproducibility for the community?

Key findings

• InternLM2 outperforms prior open-source models across comprehensive evaluations and long-context tasks.
• The model achieves strong long-context performance, including near-complete needle detection in a 200k context test.
• COOL RLHF improves subjective alignment outcomes by reconciling diverse preferences and mitigating reward hacking.
• High-quality, safety-filtered pre-training data are crucial for stability and performance.
• InternEvo enables scalable, fault-tolerant training and interactive RLHF across thousands of GPUs.