[Paper Review] TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning

Han Cai, Chuang Gan+2 more|arXiv (Cornell University)|Jul 22, 2020

Advanced Neural Network ApplicationsComputer Science61 references33 citations

TL;DR

TinyTL freezes the feature extractor weights and only trains biases, augmented by lite residual modules to refine features, achieving large memory savings (up to 12.9×) with comparable or better accuracy than full fine-tuning.

ABSTRACT

On-device learning enables edge devices to continually adapt the AI models to new data, which requires a small memory footprint to fit the tight memory constraint of edge devices. Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learns the bias modules, thus no need to store the intermediate activations. To maintain the adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 34.1%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.

Motivation & Objective

• Motivate memory-efficient on-device learning due to limited edge-device memory and energy budgets.
• Identify the bottleneck in training memory as activations rather than trainable parameters.
• Propose TinyTL to freeze weights and train biases, augmented with lite residual modules to maintain adaptation capacity.
• Evaluate memory-accuracy trade-offs across multiple datasets and backbones, including feature extractor adaptation.

Proposed method

• Analyze back-propagation memory to show activations, not weights, dominate training memory.
• Freeze feature extractor weights and train only biases to reduce activation storage.
• Introduce lite residual module to refine intermediate features with small memory overhead (~3.8%).
• Use group convolutions and controlled resolution/width to minimize activation size in the lite residuals.
• Replace BN with GN for small-batch on-device training compatibility.
• Incorporate feature extractor adaptation via Once-For-All networks to select task-specific backbones.

Experimental results

Research questions

• RQ1 Can freezing weights and training only biases substantially reduce on-device training memory without large accuracy loss?
• RQ2 Does the lite residual module provide enough capacity to compensate for frozen weights across diverse datasets?
• RQ3 How does TinyTL perform with and without feature extractor adaptation across multiple backbones?
• RQ4 What are the memory-accuracy trade-offs when applying TinyTL to varied input resolutions and batch sizes?

Key findings

• Memory footprint reduction up to 12.9× compared to fine-tuning the full network.
• TinyTL variants with lite residuals (L+B) outperform bias-only or norm-only fine-tuning baselines in accuracy.
• With higher input resolution (320), TinyTL-L+B matches full finetuning accuracy while saving ~6× memory.
• Combined with feature extractor adaptation (Once-For-All), TinyTL achieves 7.5–12.9× memory savings with comparable accuracy to fine-tuning Inception-V3.
• Training with batch size 1 further reduces memory to ~16MB, enabling SRAM-based training.
• Lite residual modules are essential for preserving adaptation capacity over bias-only approaches.