[Paper Review] How to Train Data-Efficient LLMs

Noveen Sachdeva, Benjamin Coleman+7 more|arXiv (Cornell University)|Feb 15, 2024

Advanced Data Storage Technologies4 citations

TL;DR

This paper proposes Ask-LLM and Density sampling to improve data efficiency in large language model (LLM) pre-training by prioritizing data quality and coverage, respectively. Ask-LLM uses a small instruction-tuned LLM to score training examples for quality, enabling models to outperform full-data training even when 90% of data is removed, while converging up to 70% faster.

ABSTRACT

The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.

Motivation & Objective

• Address the high computational and data cost of pre-training large language models (LLMs), which face diminishing returns due to power-law scaling.
• Investigate whether data quality estimation or coverage maximization yields better data efficiency in LLM pre-training.
• Evaluate the trade-offs between expensive quality-based sampling and cheaper coverage-based sampling in real-world LLM training.
• Determine whether data curation can improve the Pareto frontier of model quality versus data/resource consumption.
• Provide a large-scale empirical benchmark of 19 sampling strategies across multiple model sizes and evaluation tasks.

Proposed method

• Propose Ask-LLM: a data selection method that uses a zero-shot reasoning-capable instruction-tuned LLM (e.g., Flan-T5-XL) to score training examples based on their informativeness and quality.
• Implement Density sampling: a coverage-driven approach that models the latent feature distribution of data to select diverse, representative samples.
• Use perplexity-based filtering as a proxy for data quality in Ask-LLM, where lower perplexity indicates higher quality.
• Train T5-Large (800M) and T5-Small (60M) models on curated data subsets using 19 different sampling strategies.
• Evaluate all models on 111 downstream NLP tasks after fine-tuning, enabling comprehensive comparison of performance and convergence speed.
• Analyze convergence rates, sampler correlations, and the impact of sampling cost on downstream performance to isolate key drivers of data efficiency.

Figure 1 : Data-efficient pre-training run of T5-Large ( $800$ M) using Ask-LLM with Flan-T5-XL as the data quality scorer. Training on $60$ % of the original dataset, Ask-LLM is able to train T5-Large both better and $70$ % faster, compared to training on $100$ % of the dataset.

Experimental results

Research questions

• RQ1 Can quality-based data selection via LLMs outperform full-data training in terms of model performance and training speed?
• RQ2 Does coverage-based sampling (e.g., Density) recover the performance of full-data pre-training?
• RQ3 How do the trade-offs between data quality and coverage affect model performance across different model sizes and sampling rates?
• RQ4 Is the computational cost of quality estimation justified by performance gains compared to simpler coverage heuristics?
• RQ5 Can data curation techniques shift the Pareto frontier of model quality versus data/resource consumption in LLM pre-training?

Key findings

• Ask-LLM consistently outperforms full-data training, achieving better downstream performance even when 90% of the training data is removed.
• Models trained with Ask-LLM converge up to 70% faster than those trained on the full dataset, despite using significantly less data.
• Density sampling recovers the performance of full-data pre-training by maximizing coverage of latent topics in the data distribution.
• The small proxy model in Ask-LLM (60M parameters) achieves superior results compared to most established data curation baselines.
• Ask-LLM’s performance exceeds that of full-data training, indicating that data quality filtering can surpass the benefits of data quantity alone.
• The study demonstrates that data curation can shift the Pareto frontier of model quality versus data/resource use, enabling higher-quality models with less data and compute.

Figure 2 : While there is no inherent tradeoff between coverage and quality, samplers target these metrics on a spectrum (up and to the left indicates a more aggressive prioritization). See Appendix B for a description of the plotted samplers.