[Paper Review] DTL: Disentangled Transfer Learning for Visual Recognition

Minghao Fu, Ke Zhu+1 more|arXiv (Cornell University)|Dec 13, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

DTL proposes a disentangled transfer learning framework that decouples trainable parameters from the frozen backbone using a lightweight Compact Side Network (CSN), drastically reducing GPU memory usage and trainable parameters while achieving state-of-the-art accuracy on vision benchmarks. By injecting task-specific knowledge via low-rank linear mappings and selective feature calibration, DTL outperforms existing PETL methods with superior efficiency and performance.

ABSTRACT

When pre-trained models become rapidly larger, the cost of fine-tuning on downstream tasks steadily increases, too. To economically fine-tune these models, parameter-efficient transfer learning (PETL) is proposed, which only tunes a tiny subset of trainable parameters to efficiently learn quality representations. However, current PETL methods are facing the dilemma that during training the GPU memory footprint is not effectively reduced as trainable parameters. PETL will likely fail, too, if the full fine-tuning encounters the out-of-GPU-memory issue. This phenomenon happens because trainable parameters from these methods are generally entangled with the backbone, such that a lot of intermediate states have to be stored in GPU memory for gradient propagation. To alleviate this problem, we introduce Disentangled Transfer Learning (DTL), which disentangles the trainable parameters from the backbone using a lightweight Compact Side Network (CSN). By progressively extracting task-specific information with a few low-rank linear mappings and appropriately adding the information back to the backbone, CSN effectively realizes knowledge transfer in various downstream tasks. We conducted extensive experiments to validate the effectiveness of our method. The proposed method not only reduces a large amount of GPU memory usage and trainable parameters, but also outperforms existing PETL methods by a significant margin in accuracy, achieving new state-of-the-art on several standard benchmarks. The code is available at https://github.com/heekhero/DTL.

Motivation & Objective

• To address the critical limitation of existing parameter-efficient transfer learning (PETL) methods, where high GPU memory usage persists despite reduced trainable parameters.
• To reduce the GPU memory footprint during fine-tuning of large pre-trained models, which often leads to out-of-memory failures.
• To enable efficient and effective fine-tuning of large models by decoupling trainable parameters from the backbone through a disentangled architecture.
• To achieve state-of-the-art performance with minimal trainable parameters and low memory consumption across diverse vision benchmarks.

Proposed method

• Introduces a Compact Side Network (CSN) with low-rank linear projections to extract task-specific information in parallel to the backbone.
• Injects the CSN's output back into specific later blocks of the backbone (starting from index M) to adapt features without updating the backbone weights.
• Uses a disentangled design where only CSN weights and the classification head are fine-tuned, keeping the backbone frozen and eliminating gradient caching on backbone activations.
• Employs Swish activation and global depthwise separable convolution (DWConv) in DTL+ to enhance feature representation and spatial modeling.
• Supports feature reuse across tasks by freezing early backbone blocks, enabling efficient multi-task inference.
• Applies low-rank parameterization to minimize trainable parameters while maintaining high representational capacity.

Experimental results

Research questions

• RQ1 Can disentangling trainable parameters from the backbone significantly reduce GPU memory usage during fine-tuning?
• RQ2 Does a lightweight, parallel side network enable effective knowledge transfer with minimal parameter updates?
• RQ3 How does the choice of injection point (M) affect accuracy and memory efficiency in the disentangled framework?
• RQ4 Can the proposed method outperform existing PETL methods in accuracy while using fewer parameters and less memory?
• RQ5 How does the addition of DWConv and activation functions improve performance in the DTL+ variant?

Key findings

• DTL reduces GPU memory usage and trainable parameters significantly, achieving the highest accuracy on VTAB-1K with the fewest parameters and memory footprint.
• On ImageNet, ImageNet-Sketch, and ImageNet-R, DTL+ achieves an average accuracy gain of up to 8% over the previous SOTA method NOAH.
• The ablation study shows that DTL+ with d′=2 achieves 77.7% average accuracy on VTAB-1K, outperforming the baseline (76.0%) and other architectural variants.
• The method maintains high inference throughput, with DTL achieving 892 images/sec on a single 3090 GPU (batch size 16), outperforming baselines like AdaptFormer and NOAH.
• Sensitivity analysis shows that accuracy is relatively insensitive to the choice of M (injection block index), with performance stable across M=7 to M=12, indicating robustness.
• Even with d′=1 (extremely low rank), DTL+ maintains strong performance, demonstrating effectiveness under extreme parameter efficiency.