[Paper Review] ReFT: Representation Finetuning for Language Models

Zhengxuan Wu, Aryaman Arora+5 more|arXiv (Cornell University)|Apr 4, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces Representation Finetuning (ReFT), a novel parameter-efficient method that edits hidden representations in frozen language models instead of updating weights. ReFT, particularly its LoReFT variant, achieves state-of-the-art performance with 15×–65× fewer parameters than LoRA across commonsense reasoning, arithmetic, instruction tuning, and GLUE benchmarks, demonstrating that representation editing is a more efficient and effective alternative to weight-based PEFTs.

ABSTRACT

Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing a family of Representation Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x--65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE. In all these evaluations, our ReFTs deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. We release a generic ReFT training library publicly at https://github.com/stanfordnlp/pyreft.

Motivation & Objective

• To explore whether editing model representations is more effective than updating weights for fine-tuning large language models.
• To develop a family of parameter-efficient methods—ReFT—that operate on frozen base models by learning task-specific interventions on hidden representations.
• To design a highly efficient instance of ReFT, LoReFT, that uses low-rank linear subspaces to manipulate representations with minimal parameters.
• To evaluate ReFT on diverse benchmarks, including commonsense reasoning, arithmetic, instruction tuning, and GLUE, to demonstrate its superiority over existing PEFTs.
• To release a generic ReFT training library to enable broad adoption and further research in representation-based fine-tuning.

Proposed method

• ReFT operates on a frozen base language model and learns task-specific interventions on hidden representations during fine-tuning.
• The core method, LoReFT, uses a low-rank linear subspace to project and modify hidden representations, inspired by distributed alignment search (DAS).
• LoReFT parametrizes interventions as a low-rank matrix applied to hidden states, enabling efficient and effective representation editing.
• An ablation, DiReFT, further reduces parameters by using a diagonal projection matrix, trading some performance for higher efficiency.
• Interventions are trained end-to-end via standard backpropagation, with the base model weights frozen, and the learned edits applied at inference time.
• The method is a drop-in replacement for existing PEFTs like LoRA, requiring no architectural changes to the base model.

Experimental results

Research questions

• RQ1 Can editing hidden representations in frozen language models achieve better performance and efficiency than weight-based parameter-efficient fine-tuning?
• RQ2 How does representation editing compare to weight updates in terms of parameter efficiency and downstream task performance?
• RQ3 Can a low-rank linear subspace intervention on representations match or exceed the performance of state-of-the-art PEFTs like LoRA?
• RQ4 Does ReFT scale effectively with model size, particularly on larger models such as Llama-3 8B?
• RQ5 Can ReFT generalize across diverse NLP tasks, including commonsense reasoning, arithmetic, instruction tuning, and natural language understanding?

Key findings

• LoReFT achieves state-of-the-art performance on all evaluated benchmarks—commonsense reasoning, arithmetic, instruction tuning, and GLUE—while using 15×–65× fewer parameters than LoRA.
• On the LLaMA-3 8B model, LoReFT outperforms all baseline PEFTs, including LoRA, across all tasks, demonstrating superior efficiency and effectiveness.
• In commonsense reasoning tasks (e.g., BoolQ, HellaSwag), LoReFT achieves performance comparable to full fine-tuning with only 0.1% of the parameters updated.
• For arithmetic reasoning (e.g., GSM8K, AQuA), LoReFT matches or exceeds LoRA’s performance with significantly fewer trainable parameters.
• The DiReFT ablation achieves even higher efficiency with minimal performance drop, showing that representation editing can be made extremely parameter-efficient.
• The ReFT library is released publicly at https://github.com/stanfordnlp/pyreft, enabling reproducibility and future research in representation-based fine-tuning.