[Paper Review] LIMA: Less Is More for Alignment

Chunting Zhou, Pengfei Liu+13 more|arXiv (Cornell University)|May 18, 2023

Topic ModelingComputer Science126 citations

TL;DR

A 65B LLaMa model fine-tuned on 1,000 carefully curated prompts/responses (no RLHF) achieves strong alignment, often matching or exceeding baselines in human evaluations, suggesting pretraining dominates beyond small amounts of instruction tuning.

ABSTRACT

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.

Motivation & Objective

• Demonstrate that a strong pretrained language model can be effectively aligned using only 1,000 high-quality demonstrations without RLHF or human preference modeling.
• Test whether alignment primarily relies on pretraining versus instruction tuning by evaluating LIMA against state-of-the-art baselines.
• Investigate data diversity and quality versus quantity in alignment data and assess multi-turn dialogue capabilities.

Proposed method

• Fine-tune a 65B-parameter LLaMa model (LLaMa-65B) on 1,000 demonstrations (750 from community sources, 250 manually authored) using standard supervised loss.
• Introduce a special end-of-turn token to differentiate user/assistant turns during fine-tuning.
• Compare LIMA to RLHF-tuned and other baselines across 300 prompts in human preferences and GPT-4+ annotator trials.
• Conduct ablations on data diversity, quality, and quantity using a 7B model to isolate effects of each factor.
• Evaluate multi-turn dialogue capability with zero-shot and augmented dialogue-chain expansions.
• Assess safety behavior with a small set of safety-related prompts and analyze failure modes.

Figure 1 : Human preference evaluation, comparing LIMA to 5 different baselines across 300 test prompts.

Experimental results

Research questions

• RQ1 Can a pretrained LLM be effectively aligned with only 1,000 demonstrations without RLHF or preference modeling?
• RQ2 How do data diversity and quality versus sheer quantity affect alignment performance?
• RQ3 To what extent does a tiny amount of curated dialogue data improve multi-turn dialogue capabilities?
• RQ4 How does LIMA compare to state-of-the-art aligned models in human and GPT-4-based evaluations?

Key findings

• LIMA achieves competitive performance against DaVinci003 and Alpaca when evaluated by humans and by GPT-4 annotators.
• Half of LIMA outputs are rated excellent in absolute quality assessments.
• Increasing data quantity alone yields diminishing returns without increases in prompt diversity and data quality.
• Adding 30 hand-crafted dialogue chains substantially improves multi-turn dialogue quality (from 45.2% to 76.1% excellent).
• LIMA’s coherent multi-turn dialogue emerges even with zero dialogue data, but quality scales with targeted dialogue augmentation.
• On safety prompts, LIMA responds safely to 80% of cases with a small safety-focused subset in training.

Figure 2 : Preference evaluation using GPT-4 as the annotator, given the same instructions provided to humans.