[Paper Review] Composing Parameter-Efficient Modules with Arithmetic Operations

Jinghan Zhang, Shiqi Chen+2 more|arXiv (Cornell University)|Jun 26, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a training-free method to compose parameter-efficient modules (PEMs) via linear arithmetic operations—addition and negation—in the weight space, enabling flexible integration of skills from pre-trained PEMs for distribution generalization, multi-tasking, unlearning, and domain transfer. The approach achieves state-of-the-art performance across all settings without additional fine-tuning, including a 50%+ reduction in toxicity when detoxifying Alpaca-LoRA.

ABSTRACT

As an efficient alternative to conventional full finetuning, parameter-efficient finetuning (PEFT) is becoming the prevailing method to adapt pretrained language models. In PEFT, a lightweight module is learned on each dataset while the underlying pretrained language model remains unchanged, resulting in multiple compact modules representing diverse skills when applied to various domains and tasks. In this paper, we propose to compose these parameter-efficient modules through linear arithmetic operations in the weight space, thereby integrating different module capabilities. Specifically, we first define addition and negation operators for the module, and then further compose these two basic operators to perform flexible arithmetic. Our approach requires \emph{no additional training} and enables highly flexible module composition. We apply different arithmetic operations to compose the parameter-efficient modules for (1) distribution generalization, (2) multi-tasking, (3) unlearning, and (4) domain transfer. Additionally, we extend our approach to detoxify Alpaca-LoRA, the latest instruction-tuned large language model based on LLaMA. Empirical results demonstrate that our approach produces new and effective parameter-efficient modules that significantly outperform existing ones across all settings.

Motivation & Objective

• To enable flexible, training-free composition of pre-trained parameter-efficient modules (PEMs) for diverse downstream tasks.
• To address the challenge of combining PEMs with different capabilities—such as multi-tasking, distribution generalization, unlearning, and domain transfer—without retraining.
• To extend PEM composition to modern large language models like LLaMA, including practical applications such as model detoxification.
• To investigate whether linear arithmetic operations on PEM parameters can yield superior performance compared to existing composition methods.
• To evaluate the robustness of PEM composition under varying initialization schemes and hyperparameters.

Proposed method

• Define addition and negation operators for LoRA and (IA)3 PEM architectures to enable linear arithmetic composition in parameter space.
• Compose PEMs using weighted linear combinations (e.g., α·PEM₁ + β·PEM₂) to merge skills, with β = -1 for unlearning.
• Apply the composition operation directly on the learned adapter matrices without retraining the base model or the PEMs.
• Use a hyperparameter λ to control the balance between source PEMs, tuned via validation performance.
• Extend the method to detoxify Alpaca-LoRA by composing a toxic PEM with a neutral one to reduce harmful outputs.
• Validate the approach across four settings: distribution generalization, multi-tasking, unlearning, and domain transfer using standard NLP benchmarks.

Experimental results

Research questions

• RQ1 Can linear arithmetic operations on PEM parameters enable effective composition of diverse model capabilities without retraining?
• RQ2 How does PEM composition via addition and negation perform in multi-task learning and distribution generalization compared to baseline methods?
• RQ3 Can PEM composition be used to unlearn specific model behaviors, such as generating toxic outputs?
• RQ4 How robust is PEM composition to variations in model initialization and hyperparameters?
• RQ5 Can PEM composition effectively detoxify a large instruction-tuned LLM like Alpaca-LoRA while preserving performance on non-toxic inputs?

Key findings

• The proposed method achieves significant performance gains across all four settings—distribution generalization, multi-tasking, unlearning, and domain transfer—without any additional training.
• In domain transfer, the merged PEM outperformed individual PEMs on both Yelp and Amazon datasets, with average improvements of +1.65% (same initialization) and +1.18% (different initialization).
• For detoxification of Alpaca-LoRA, the merged PEM reduced toxicity scores by over 50% (from 0.321 to 0.158) while maintaining or improving helpfulness and win rates.
• Manual evaluation showed a 36% win rate on toxic instructions and 27% on normal ones, with high tie rates (40% and 42%), indicating strong alignment with human preferences.
• Even when PEMs were initialized with different random seeds, the merged model retained strong performance, indicating robustness to initialization variance.
• The method successfully preserved performance on non-toxic inputs while significantly reducing toxicity, demonstrating effective unlearning of harmful behaviors.