[論文レビュー] What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

Wei Liu, Weihao Zeng+3 more|arXiv (Cornell University)|Dec 25, 2023

Topic Modeling被引用数 14

ひとこと要約

本論文は instruction tuning のデータ品質を分析し、複雑さ・品質・多様性を用いて tens of thousands のデータポイントと強い整合を達成する、データ効率的なデータ選択手法 Deita を提案します。

ABSTRACT

Instruction tuning is a standard technique employed to align large language models to end tasks and user preferences after the initial pretraining phase. Recent research indicates the critical role of data engineering in instruction tuning -- when appropriately selected, only limited data is necessary to achieve superior performance. However, we still lack a principled understanding of what makes good instruction tuning data for alignment, and how we should select data automatically and effectively. In this work, we delve deeply into automatic data selection strategies for alignment. We start with controlled studies to measure data across three dimensions: complexity, quality, and diversity, along which we examine existing methods and introduce novel techniques for enhanced data measurement. Subsequently, we propose a simple strategy to select data samples based on the measurement. We present deita (short for Data-Efficient Instruction Tuning for Alignment), a series of models fine-tuned from LLaMA and Mistral models using data samples automatically selected with our proposed approach. Empirically, deita performs better or on par with the state-of-the-art open-source alignment models with only 6K SFT training data samples -- over 10x less than the data used in the baselines. When further trained with direct preference optimization (DPO), deita-Mistral-7B + DPO trained with 6K SFT and 10K DPO samples achieve 7.55 MT-Bench and 90.06% AlpacaEval scores. We anticipate this work to provide tools on automatic data selection, facilitating data-efficient alignment. We release our models as well as the selected datasets for future researches to effectively align models more efficiently.

研究の動機と目的

• instruction tuning across three dimensions: complexity, quality, and diversity define what constitutes good data.
• Develop automatic measurement techniques for each dimension and evaluate their correlation with alignment performance.
• Propose a simple, practical data selection strategy that combines these measurements to maximize data efficiency.
• Demonstrate the effectiveness of Deita on LLaMA-1/2 and Mistral backbones against open-source alignment models.
• Release data and model checkpoints to facilitate future research in data-efficient alignment.

提案手法

• Introduce three data-diagnostic dimensions: complexity, quality, and diversity, and design evaluation metrics for each.
• Use Evol Complexity and Evol Quality to automatically generate and score data variants via an evolution-based process.
• Train lightweight scorers to predict complexity/quality scores from instructions; combine scores into an evol score.
• Apply a score-first, diversity-aware selection strategy to pick a small, high-quality, diverse subset from large data pools.
• Fine-tune LLaMA-1-13B, LLaMA-2-13B, and Mistral-7B with the selected data to obtain Deita models; optionally apply Direct Preference Optimization (DPO).
• Compare Deita against baseline data-selection approaches and open-source SFT models on MT-Bench, AlpacaEval, and Open LLM Leaderboard.

実験結果

リサーチクエスチョン

• RQ1 What data properties (complexity, quality, diversity) most strongly correlate with instruction-following alignment performance?
• RQ2 Can automatic, scalable metrics for complexity and quality replace expensive human or GPT-based annotations for data selection?
• RQ3 Does a score-first, diversity-aware data selection strategy yield data-efficient alignment across different base models and data pools?
• RQ4 How do Deita models perform compared with state-of-the-art open-source alignment models when trained with only a small data budget (6K–10K)?
• RQ5 What is the impact of applying DPO on top of the Deita-selected data?

主な発見

• Evol Complexity consistently yields superior MT-Bench performance across data pools, outperforming baselines like Random and Instag Diversity.
• Evol Quality consistently improves MT-Bench performance, especially on Lew-rich low-quality pools, and shows notable gains with higher quality variance.
• Embedding-based diversity (Repr Filter) outperforms Instag Diversity, highlighting the importance of diverse representations.
• Deita models trained with 6K–10K data achieve competitive or superior MT-Bench and AlpacaEval scores compared to larger SFT baselines; Deita-Mistral-7B 10K with DPO achieves 7.55 MT-Bench and 90.06% AlpacaEval.
• On Open LLM Leaderboard, Deita variants consistently outperform many SFT models, and DPO further boosts performance.
• Training with Deita can reach near-parity with much larger data usage, showing a 100x data reduction feasibility for competitive alignment.