[Paper Review] K for the price of 1. Parameter efficient multi-task and transfer learning

Pramod Kaushik Mudrakarta, M. Sandler+2 more|arXiv (Cornell University)|Sep 27, 2018

Domain Adaptation and Few-Shot Learning19 citations

TL;DR

This paper proposes a parameter-efficient method for multi-task and transfer learning by learning task-specific 'model patches'—small sets of parameters like scales and biases—instead of fine-tuning entire networks. By reusing 98% of a pretrained SSD feature extractor and only adapting a small parameter set, the approach achieves performance comparable to full fine-tuning while significantly reducing parameter updates.

ABSTRACT

We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that learning a set of scales and biases is sufficient to convert a pretrained network to perform well on qualitatively different problems (e.g. converting a Single Shot MultiBox Detection (SSD) model into a 1000-class image classification model while reusing 98% of parameters of the SSD feature extractor). Similarly, we show that re-learning existing low-parameter layers (such as depth-wise convolutions) while keeping the rest of the network frozen also improves transfer-learning accuracy significantly. Our approach allows both simultaneous (multi-task) as well as sequential transfer learning. In several multi-task learning problems, despite using much fewer parameters than traditional logits-only fine-tuning, we match single-task performance.

Motivation & Objective

• To address the inefficiency of full fine-tuning in transfer and multi-task learning by reducing the number of trainable parameters.
• To enable both sequential and simultaneous transfer learning without retraining the entire network.
• To maintain high performance on downstream tasks while reusing the majority of pretrained weights.
• To explore whether small, learnable parameter patches can effectively adapt a single backbone to diverse tasks.

Proposed method

• The method introduces task-specific model patches consisting of a small set of learnable parameters, such as scales and biases, that are added to a frozen pretrained network.
• These patches are trained to adapt the pretrained features to specific tasks, enabling parameter-efficient adaptation.
• The approach supports both multi-task learning (simultaneous adaptation to multiple tasks) and sequential transfer learning (one task after another).
• The method allows re-learning low-parameter layers (e.g., depthwise convolutions) while keeping the rest of the network frozen, improving transfer accuracy.
• The model patches are applied after the feature extractor, modifying the feature representations before the final classification head.
• The approach maintains the original network architecture and only introduces a minimal number of additional parameters per task.

Experimental results

Research questions

• RQ1 Can a small set of learnable parameters effectively adapt a frozen pretrained network to diverse downstream tasks?
• RQ2 How does the performance of the proposed method compare to full fine-tuning and logits-only adaptation in multi-task learning?
• RQ3 To what extent can the same feature extractor be reused across multiple tasks with minimal parameter updates?
• RQ4 Does re-learning low-parameter layers while keeping the rest frozen improve transfer learning accuracy?
• RQ5 Can the method achieve single-task performance levels while using significantly fewer parameters than standard fine-tuning?

Key findings

• The method achieves performance comparable to full fine-tuning while reusing 98% of the parameters from a pretrained SSD feature extractor.
• The approach matches single-task performance in multi-task learning despite using far fewer trainable parameters than standard fine-tuning.
• Re-learning low-parameter layers (e.g., depthwise convolutions) while keeping the rest frozen improves transfer learning accuracy.
• The model patches enable effective adaptation across qualitatively different tasks, such as converting an object detection model into a 1000-class image classifier.
• The method supports both simultaneous and sequential transfer learning with minimal parameter updates.
• The approach demonstrates strong parameter efficiency, making it suitable for deployment in resource-constrained settings.