[Paper Review] Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

Mayee F. Chen, Nicholas Roberts+5 more|arXiv (Cornell University)|Jul 26, 2023

Topic Modeling4 citations

TL;DR

This paper proposes Skill-it, a data-driven framework that models language model (LM) training as a sequence of interdependent skills, formalized through a skill graph to guide efficient data sampling. By exploiting skill ordering in continual pre-training and fine-tuning, Skill-it achieves 36.5 higher accuracy than random sampling on LEGO and reduces validation loss by 13.6% on Natural Instructions, outperforming static and baseline sampling methods.

ABSTRACT

The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good downstream model performance across tasks. We develop a new framework based on a simple hypothesis: just as humans acquire interdependent skills in a deliberate order, language models also follow a natural order when learning a set of skills from their training data. If such an order exists, it can be utilized for improved understanding of LMs and for data-efficient training. Using this intuition, our framework formalizes the notion of a skill and of an ordered set of skills in terms of the associated data. First, using both synthetic and real data, we demonstrate that these ordered skill sets exist, and that their existence enables more advanced skills to be learned with less data when we train on their prerequisite skills. Second, using our proposed framework, we introduce an online data sampling algorithm, Skill-It, over mixtures of skills for both continual pre-training and fine-tuning regimes, where the objective is to efficiently learn multiple skills in the former and an individual skill in the latter. On the LEGO synthetic in the continual pre-training setting, Skill-It obtains 36.5 points higher accuracy than random sampling. On the Natural Instructions dataset in the fine-tuning setting, Skill-It reduces the validation loss on the target skill by 13.6% versus training on data associated with the target skill itself. We apply our skills framework on the recent RedPajama dataset to continually pre-train a 3B-parameter LM, achieving higher accuracy on the LM Evaluation Harness with 1B tokens than the baseline approach of sampling uniformly over data sources with 3B tokens.

Motivation & Objective

• To develop a formal framework for modeling language model capabilities as interdependent skills derived from data.
• To investigate whether a natural learning order of skills exists in training data and can be leveraged for data-efficient training.
• To design a dynamic, online data sampling algorithm that exploits skill dependencies to accelerate learning in continual pre-training and fine-tuning.
• To validate that skill-based ordering improves model performance compared to random or static sampling strategies.
• To demonstrate the framework's effectiveness on synthetic and real-world datasets, including RedPajama and Natural Instructions.

Proposed method

• Define a skill as a unit of behavior learnable from a data slice, and formalize an ordered skill set via a directed graph where edges indicate prerequisite dependencies.
• Construct a skill graph using data clustering and embedding analysis to identify skill relationships, validated on synthetic and real datasets.
• Design Skill-it, an online data selection algorithm that dynamically adjusts sampling weights based on skill validation loss and graph structure.
• Use multiplicative weight updates with a learning rate schedule to prioritize harder skills while respecting prerequisite dependencies.
• Integrate the skill graph into both continual pre-training and fine-tuning regimes to guide data mixture selection.
• Evaluate the framework using ablation studies comparing Skill-it to random sampling, no graph, and static sampling (T=1) across multiple benchmarks.

Experimental results

Research questions

• RQ1 Do interdependent skill sets exist in real and synthetic language model training data, and can they be formally defined using data-based clustering and embeddings?
• RQ2 Can training on prerequisite skills reduce the data needed to learn more complex skills, and does this lead to measurable performance gains?
• RQ3 How does Skill-it’s dynamic, graph-aware sampling strategy compare to random and static sampling in terms of convergence speed and final performance?
• RQ4 To what extent does the skill graph structure and online adaptation of sampling weights improve learning efficiency in continual pre-training and fine-tuning?
• RQ5 Can the framework be generalized to real-world datasets like RedPajama and Natural Instructions, and does it outperform uniform sampling with the same token budget?

Key findings

• On the LEGO synthetic continual pre-training task, Skill-it achieved 36.5 points higher accuracy than random sampling, demonstrating significant performance gains from skill ordering.
• On the Natural Instructions dataset, Skill-it reduced validation loss on the target skill by 13.6% compared to training only on data associated with the target skill.
• Skill-it outperformed static sampling (T=1) and no-graph baselines across all evaluated datasets, including LEGO, Addition, and Natural Instructions, due to its dynamic adaptation and graph-aware weighting.
• In the out-of-domain evaluation on Natural Instructions, Skill-it achieved the lowest validation loss on 7 out of 12 skills and had the lowest average loss (2.540) compared to 2.541–2.551 for static methods.
• Ablation studies confirmed that both the skill graph and online dynamics are essential: removing either leads to suboptimal allocation of training tokens, especially to harder skills.
• The framework successfully enabled a 3B-parameter LM to achieve higher accuracy on the LM Evaluation Harness when continually pre-trained on RedPajama using 1B tokens, outperforming uniform sampling with 3B tokens.