[Paper Review] PAFT: A Parallel Training Paradigm for Effective LLM Fine-Tuning

Shiva Kumar Pentyala, Zhichao Wang+7 more|arXiv (Cornell University)|Jun 25, 2024

Advanced Materials Characterization Techniques4 citations

TL;DR

PAFT introduces a parallel training paradigm that independently performs supervised fine-tuning (SFT) and preference alignment (e.g., DPO) on the same pre-trained LLM, then merges the resulting models via parameter fusion. By applying L1 regularization to SFT to induce sparsity, PAFT reduces interference during merging, achieving SOTA performance—ranking #1 on the HuggingFace Open LLM Leaderboard with a 0.6524 average score on the 7B model category.

ABSTRACT

Large language models (LLMs) have shown remarkable abilities in diverse natural language processing (NLP) tasks. The LLMs generally undergo supervised fine-tuning (SFT) followed by preference alignment to be usable in downstream applications. However, this sequential training pipeline leads to alignment tax that degrades the LLM performance. This paper introduces PAFT, a new PArallel training paradigm for effective LLM Fine-Tuning, which independently performs SFT and preference alignment (e.g., DPO and ORPO, etc.) with the same pre-trained model on respective datasets. The model produced by SFT and the model from preference alignment are then merged into a final model by parameter fusing for use in downstream applications. This work reveals important findings that preference alignment like DPO naturally results in a sparse model while SFT leads to a natural dense model which needs to be sparsified for effective model merging. This paper introduces an effective interference resolution which reduces the redundancy by sparsifying the delta parameters. The LLM resulted from the new training paradigm achieved Rank #1 on the HuggingFace Open LLM Leaderboard. Comprehensive evaluation shows the effectiveness of the parallel training paradigm.

Motivation & Objective

• To address the alignment tax caused by sequential SFT followed by preference alignment in LLM fine-tuning.
• To reduce performance degradation during model merging due to parameter interference from redundant delta parameters.
• To improve model merging effectiveness by inducing sparsity in SFT delta parameters through L1 regularization.
• To demonstrate that parallel training with sparse model integration outperforms sequential training and existing merging techniques.
• To establish a robust, generalizable framework for combining SFT and preference alignment that achieves state-of-the-art results on public benchmarks.

Proposed method

• Proposes a parallel training paradigm (PAFT) that independently trains a model for SFT and another for preference alignment (e.g., DPO) on separate datasets.
• Applies an L1-norm penalty to the SFT loss function to implicitly induce sparsity in the delta parameters (differences between fine-tuned and pre-trained weights).
• Uses sparse delta parameters from SFT and preference alignment for model merging via parameter fusion to preserve model capabilities.
• Employs TIES and Task Arithmetic as top-performing merging methods, with TIES showing superior performance on Mistral-7B.
• Validates the method using comprehensive evaluation on HuggingFace Open LLM Leaderboard and AlpacaEval benchmarks.
• Compares the L1-based sparsity method with explicit sparsity techniques like DARE, showing L1's superiority in performance and quality.

(a) Staged training

Experimental results

Research questions

• RQ1 Does parallel training of SFT and preference alignment reduce the alignment tax compared to sequential training?
• RQ2 Can inducing sparsity in SFT delta parameters mitigate interference during model merging?
• RQ3 Is L1 regularization more effective than explicit sparsity methods like DARE for improving merged model performance?
• RQ4 Which model merging technique (e.g., TIES, Task Arithmetic) yields the best results when combined with PAFT?
• RQ5 Can PAFT generalize across different preference alignment methods, such as ORPO, beyond DPO?

Key findings

• The PAFT paradigm achieved a new SOTA score of 0.6524 on average across six tasks in the HuggingFace Open LLM Leaderboard for 7B models, ranking #1 in the 7B/8B category.
• The PAFT-ed 70B model achieved the top global rank on the HuggingFace Open LLM Leaderboard, demonstrating scalability and generalization.
• Parallel SFT with L1-regularized sparsity (Parallel SFT_spar se) consistently outperformed non-sparse SFT across all merging methods and benchmarks.
• The L1-norm method for inducing sparsity in SFT delta parameters was found to be more effective and higher-quality than the explicit DARE method.
• TIES and Task Arithmetic were identified as the most effective merging strategies, with TIES yielding the best results on Mistral-7B.
• Preference alignment methods like DPO and ORPO both benefited from PAFT, confirming the framework’s generalizability across alignment techniques.

(b) Parallel training