[Paper Review] On the Transferability of Representations in Neural Networks Between Datasets and Tasks

Haytham M. Fayek, Lawrence Cavedon+1 more|arXiv (Cornell University)|Nov 29, 2018

Domain Adaptation and Few-Shot Learning14 references4 citations

TL;DR

This paper investigates layer-wise transferability of deep neural network representations across diverse datasets and tasks, including image recognition (CIFAR-10, CIFAR-100, SVHN) and speech tasks (ASR on TIMIT, SER on IEMOCAP). Using gradual transfer learning across multiple ConvNet architectures, it finds that transferability is non-symmetric and primarily driven by dataset/task similarity rather than network architecture, with early layers showing higher transferability than deeper layers.

ABSTRACT

Deep networks, composed of multiple layers of hierarchical distributed representations, tend to learn low-level features in initial layers and transition to high-level features towards final layers. Paradigms such as transfer learning, multi-task learning, and continual learning leverage this notion of generic hierarchical distributed representations to share knowledge across datasets and tasks. Herein, we study the layer-wise transferability of representations in deep networks across a few datasets and tasks and note some interesting empirical observations.

Motivation & Objective

• To analyze how representations learned in deep networks transfer across different datasets and tasks.
• To investigate whether transferability is symmetric or asymmetric between datasets and tasks.
• To evaluate the influence of dataset/task similarity versus neural network architecture on transferability.
• To explore the use of layer-wise transferability as a proxy for measuring task relatedness.
• To assess transferability across both computer vision and speech processing tasks using multiple architectures.

Proposed method

• Gradual transfer learning is used, where features from a pre-trained model on one dataset/task are transferred to a second dataset/task by freezing increasing numbers of initial layers.
• For each dataset/task pair, the first $ l_c \in \{0,\dots,L_H\} $ layers are held constant while the remaining layers are fine-tuned.
• Two ConvNet architectures—Model A (standard) and Model B (VGGNet variant)—are used to assess architectural robustness of findings.
• Experiments are conducted on image datasets (CIFAR-10, CIFAR-100, SVHN) and speech datasets (TIMIT for ASR, IEMOCAP for SER) with standardized preprocessing and training protocols.
• Performance is evaluated by measuring classification accuracy as a function of the number of frozen layers, indicating transferability at each layer.
• Cross-validation and standardized data splits (e.g., 8-fold leave-one-speaker-out for IEMOCAP) ensure robust evaluation.

Experimental results

Research questions

• RQ1 Is the transferability of representations between two datasets or tasks symmetric, or does one direction benefit more than the other?
• RQ2 How does the nature of the datasets or tasks influence layer-wise transferability compared to neural network architecture?
• RQ3 Can layer-wise transferability serve as a reliable proxy for measuring task relatedness?
• RQ4 How does transferability vary across different neural network architectures when applied to the same task pairs?
• RQ5 What is the pattern of transferability across layers in deep networks for both vision and speech tasks?

Key findings

• Transferability between datasets is non-symmetric: representations from CIFAR-10 and CIFAR-100 are more transferable to SVHN than vice versa, due to the general nature of CIFAR classes compared to digit-specific SVHN.
• The nature of the datasets or tasks is a stronger determinant of transferability than the neural network architecture, as both Model A and Model B show similar transferability trends.
• Early layers in deep networks exhibit significantly higher transferability than deeper layers, consistent across all tested datasets and architectures.
• Transferability decreases gradually with depth, indicating a hierarchical feature progression from generic to task-specific representations.
• The layer-wise transferability pattern is consistent across different tasks—image recognition and speech processing—suggesting a generalizable behavior in deep representations.
• The transferability proxy effectively captures task relatedness, with higher transferability observed between more similar tasks (e.g., CIFAR-10 and CIFAR-100) than dissimilar ones (e.g., SVHN and CIFAR-10).