[Paper Review] PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniques

Mohammed Sabry, Anja Belz|arXiv (Cornell University)|Apr 24, 2023

Topic Modeling4 citations

TL;DR

This paper introduces PEFT-Ref, a modular reference architecture and typology that standardizes the structural and functional properties of parameter-efficient fine-tuning (PEFT) techniques. By isolating differences in module connectivity, insertion positions, and composition, it enables systematic comparison of PEFT methods in terms of efficiency, performance, and reusability across downstream tasks.

ABSTRACT

Recent parameter-efficient finetuning (PEFT) techniques aim to improve over the considerable cost of fully finetuning large pretrained language models (PLM). As different PEFT techniques proliferate, it is becoming difficult to compare them, in particular in terms of (i) the structure and functionality they add to the PLM, (ii) the different types and degrees of efficiency improvements achieved, (iii) performance at different downstream tasks, and (iv) how differences in structure and functionality relate to efficiency and task performance. To facilitate such comparisons, this paper presents a reference architecture which standardises aspects shared by different PEFT techniques, while isolating differences to specific locations and interactions with the standard components. Through this process of standardising and isolating differences, a modular view of PEFT techniques emerges, supporting not only direct comparison of different techniques and their efficiency and task performance, but also systematic exploration of reusability and composability of the different types of finetuned modules. We demonstrate how the reference architecture can be applied to understand properties and relative advantages of PEFT techniques, hence to inform selection of techniques for specific tasks, and design choices for new PEFT techniques.

Motivation & Objective

• To address the growing difficulty in comparing diverse PEFT techniques due to inconsistent structures and evaluation metrics.
• To standardize shared components across PEFT methods while isolating differences to enable direct comparison.
• To develop a modular typology that captures structural and functional properties of PEFT modules for improved reusability and composability.
• To guide the selection of PEFT techniques for specific downstream tasks based on their architectural and functional characteristics.
• To inform the design of new PEFT methods by identifying key modularity factors that influence efficiency and performance.

Proposed method

• Proposes a modular reference architecture that decomposes PEFT techniques into standard components: a pretrained language model (PLM) and insertable PEFT modules.
• Defines seven core modular properties: intra-connectivity, inter-connectivity, insertion position, composition function, functional form, input type, and modified representation.
• Uses the reference architecture to characterize seven leading PEFT techniques (e.g., LoRA, Adapter, Prefix Tuning) along these standardized properties.
• Applies the typology to compare PEFT methods across efficiency (parameter count, FLOPs) and performance (downstream task accuracy).
• Introduces a functional form abstraction that treats PEFT modules as Project-down → Nonlinear/Linear → Project-up, while distinguishing input types and modified representations.
• Emphasizes modularity by separating the PLM’s internal workspace from the modified representations, avoiding misclassification of positions as representations.

Experimental results

Research questions

• RQ1 How can PEFT techniques be systematically compared in terms of their structural and functional differences?
• RQ2 Which architectural and functional properties of PEFT modules most strongly correlate with efficiency and downstream performance?
• RQ3 To what extent can PEFT techniques be composed or reused across different tasks and models?
• RQ4 How do differences in insertion position, connectivity, and composition affect the trade-off between parameter efficiency and model performance?
• RQ5 What standardization framework enables clearer design choices and future development of improved PEFT methods?

Key findings

• The PEFT-Ref framework enables direct comparison of PEFT techniques by standardizing shared components and isolating differences in module structure and function.
• Intra-connectivity and inter-connectivity types (e.g., dense vs. sparse) significantly influence modularity and efficiency, with most PEFT methods using dense connectivity.
• Insertion position within the Transformer architecture (e.g., after attention or feed-forward layers) affects performance and parameter efficiency, with some positions showing better generalization.
• Not all PEFT methods follow the Project-down → Nonlinear/Linear → Project-up functional form, indicating limitations in prior unifying views.
• The framework reveals that methods like Prompt Tuning and (IA)³ do not rely on residual addition, challenging assumptions about standard integration patterns.
• The typology supports systematic exploration of composability and reusability, offering a foundation for future modular PEFT design and selection in real-world applications.