[Paper Review] Entropy Law: The Story Behind Data Compression and LLM Performance

Mingjia Yin, Chuhan Wu+8 more|arXiv (Cornell University)|Jul 9, 2024

Financial Distress and Bankruptcy Prediction6 citations

TL;DR

The paper introduces an entropy law linking LLM performance to data compression ratio and first-epoch training loss, and proposes ZIP, a model-free data selection method that minimizes compression to maximize information gain.

ABSTRACT

Data is the cornerstone of large language models (LLMs), but not all data is useful for model learning. Carefully selected data can better elicit the capabilities of LLMs with much less computational overhead. Most methods concentrate on evaluating the quality of individual samples in data selection, while the combinatorial effects among samples are neglected. Even if each sample is of perfect quality, their combinations may be suboptimal in teaching LLMs due to their intrinsic homogeneity or contradiction. In this paper, we aim to uncover the underlying relationships between LLM performance and data selection. Inspired by the information compression nature of LLMs, we uncover an ``entropy law'' that connects LLM performance with data compression ratio and first-epoch training loss, which reflect the information redundancy of a dataset and the mastery of inherent knowledge encoded in this dataset, respectively. Through both theoretical deduction and empirical evaluation, we find that model performance is negatively correlated to the compression ratio of training data, which usually yields a lower training loss. Based on the findings of the entropy law, we propose a quite efficient and universal data selection method named extbf{ZIP} for training LLMs, which aim to prioritize data subsets exhibiting a low compression ratio. Based on a multi-stage algorithm that selects diverse data in a greedy manner, we can obtain a good data subset with satisfactory diversity. Extensive experiments have been conducted to validate the entropy law and the superiority of ZIP across different LLM backbones and alignment stages. We also present an interesting application of entropy law that can detect potential performance risks at the beginning of model training.

Motivation & Objective

• Motivate data selection for LLM training under limited compute by examining data information content beyond sample-level quality.
• Theoretically connect LLM learning to information compression and data redundancy through an entropy law.
• Propose an efficient, universal data subset selection method (ZIP) guided by compression ratio to maximize effective information.
• Validate entropy law and ZIP across multiple LLM backbones and alignment stages.

Proposed method

• Define data compression ratio as pre- to post-compression bits to measure data density.
• Derive an entropy law linking compression ratio, training loss, data consistency, and average data quality to model performance.
• Propose ZIP: a multi-stage greedy algorithm with global, coarse-grained local, and fine-grained local steps to select low-compression, diverse data subsets.
• Implement ZIP in SFT and RLHF experiments using CPU-only, model-free data selection, avoiding reliance on proprietary evaluators.
• Evaluate ZIP against heuristic and model-based baselines on Mistral-7B and Llama-3-8B, using MT-bench as the evaluation metric.
• Provide empirical validation of the entropy law through compression ratio and training loss analyses.

Figure 1 : An illustrative example describing different data selection paradigms. Quality-based data selection relies on sample-level data quality measurements while overlooking combinatorial effects among samples. Information-amount-based selection aims to select samples maximizing the overall info

Experimental results

Research questions

• RQ1 Does model performance correlate negatively with data compression ratio when sample quality is held constant?
• RQ2 Can a data subset with low compression ratio (high information density) improve LLM learning more than high-quality but potentially redundant samples?
• RQ3 Is ZIP effective across different LLM backbones and alignment stages (SFT and RLHF) compared to existing baselines?
• RQ4 Can entropy-law insights predict early-stage performance risks during model training?

Key findings

• Model performance is negatively correlated with the data compression ratio of the training set.
• ZIP outperforms baselines across Mistral-7B and Llama-3-8B in SFT and RLHF.
• ZIP is model-free and CPU-efficient, selecting shorter, less redundant samples with comparable or better MT-bench scores.
• Entropy law curves show lower compression ratio often aligns with better performance, even when training loss varies.
• The law enables early detection of potential performance risks during incremental training data updates.
• The data quality score of selected subsets can be similar across methods, suggesting compression-based selection captures complementary information not reflected by sample-level quality alone.

(a) ZIP