[Paper Review] MoDS: Model-oriented Data Selection for Instruction Tuning

Qianlong Du, Chengqing Zong+1 more|arXiv (Cornell University)|Nov 27, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes MoDS, a model-oriented data selection framework for instruction tuning that selects high-quality, diverse, and necessary instruction data by integrating quality, coverage, and necessity criteria. Using a quality evaluation model, k-center greedy clustering for coverage, and a necessity evaluation model based on model performance gaps, MoDS selects only 4,000 instruction pairs that outperform fine-tuning with the full 214k dataset.

ABSTRACT

Instruction tuning has become the de facto method to equip large language models (LLMs) with the ability of following user instructions. Usually, hundreds of thousands or millions of instruction-following pairs are employed to fine-tune the foundation LLMs. Recently, some studies show that a small number of high-quality instruction data is enough. However, how to select appropriate instruction data for a given LLM is still an open problem. To address this problem, in this paper we present a model-oriented data selection (MoDS) approach, which selects instruction data based on a new criteria considering three aspects: quality, coverage and necessity. First, our approach utilizes a quality evaluation model to filter out the high-quality subset from the original instruction dataset, and then designs an algorithm to further select from the high-quality subset a seed instruction dataset with good coverage. The seed dataset is applied to fine-tune the foundation LLM to obtain an initial instruction-following LLM. Finally, we develop a necessity evaluation model to find out the instruction data which are performed badly in the initial instruction-following LLM and consider them necessary instructions to further improve the LLMs. In this way, we can get a small high-quality, broad-coverage and high-necessity subset from the original instruction datasets. Experimental results show that, the model fine-tuned with 4,000 instruction pairs selected by our approach could perform better than the model fine-tuned with the full original dataset which includes 214k instruction data.

Motivation & Objective

• To address the open problem of selecting optimal instruction data for fine-tuning large language models (LLMs).
• To develop a data selection method that considers not only data quality and diversity but also the specific capability gaps of a target LLM.
• To reduce reliance on massive instruction datasets by identifying a minimal, high-impact subset of instruction-following pairs.
• To improve instruction-following performance with fewer, more strategically selected data points.

Proposed method

• A quality evaluation model scores all (instruction, input, output) triplets in the dataset to filter out high-quality examples.
• A k-center greedy algorithm selects a diverse subset from the high-quality data to maximize coverage of the instruction space.
• The selected seed dataset is used to fine-tune the target LLM, creating an initial instruction-following model.
• A necessity evaluation model identifies instructions on which the fine-tuned model performs poorly, indicating capability gaps.
• The model’s inference results on high-quality data are used to detect underperforming instructions, which are then collected as augmented data.
• The final dataset combines the seed and augmented data, forming a high-quality, broad-coverage, and high-necessity subset for re-fine-tuning.

Experimental results

Research questions

• RQ1 Can a model-oriented data selection strategy that integrates quality, coverage, and necessity criteria outperform traditional data selection methods in instruction tuning?
• RQ2 Is it possible to achieve better instruction-following performance with a small, carefully selected subset of instruction data compared to full-dataset fine-tuning?
• RQ3 How does the inclusion of necessity-aware data—selected based on model performance gaps—affect the final model’s instruction-following capability?
• RQ4 What is the optimal size of augmented data needed to effectively close capability gaps in a fine-tuned LLM?

Key findings

• The model fine-tuned with 4,000 instruction pairs selected by MoDS outperforms the model fine-tuned with the full 214k instruction dataset.
• The k-center greedy algorithm significantly improves performance over random sampling, demonstrating its effectiveness in selecting diverse, high-coverage data.
• The inclusion of augmented data—selected based on model performance gaps—leads to substantial improvements over using only the seed dataset.
• The winning score of the MoDS model exceeds 1.0 when using 3,000 augmented data points, indicating superior performance compared to the full-dataset baseline.
• Performance gains plateau when the augmented dataset reaches 3,000 examples, suggesting diminishing returns beyond this point.