[Paper Review] Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learning

Kshitij Dwivedi, Gemma Roig|arXiv (Cornell University)|Apr 26, 2019

Domain Adaptation and Few-Shot Learning34 references18 citations

TL;DR

This paper proposes a representation similarity analysis (RSA) method to efficiently rank pre-trained models for transfer learning by computing correlation-based similarity scores between their learned representations using only a few images and no further training. The key contribution is that RSA similarity strongly predicts transfer learning performance—validated on Taskonomy and Pascal VOC, where models with higher RSA scores outperform those from the same task due to domain shifts.

ABSTRACT

Transfer learning is widely used in deep neural network models when there are few labeled examples available. The common approach is to take a pre-trained network in a similar task and finetune the model parameters. This is usually done blindly without a pre-selection from a set of pre-trained models, or by finetuning a set of models trained on different tasks and selecting the best performing one by cross-validation. We address this problem by proposing an approach to assess the relationship between visual tasks and their task-specific models. Our method uses Representation Similarity Analysis (RSA), which is commonly used to find a correlation between neuronal responses from brain data and models. With RSA we obtain a similarity score among tasks by computing correlations between models trained on different tasks. Our method is efficient as it requires only pre-trained models, and a few images with no further training. We demonstrate the effectiveness and efficiency of our method for generating task taxonomy on Taskonomy dataset. We next evaluate the relationship of RSA with the transfer learning performance on Taskonomy tasks and a new task: Pascal VOC semantic segmentation. Our results reveal that models trained on tasks with higher similarity score show higher transfer learning performance. Surprisingly, the best transfer learning result for Pascal VOC semantic segmentation is not obtained from the pre-trained model on semantic segmentation, probably due to the domain differences, and our method successfully selects the high performing models.

Motivation & Objective

• To address the challenge of selecting optimal pre-trained models for transfer learning without costly fine-tuning.
• To develop an efficient method for estimating task relationships between visual tasks using only pre-trained models and a few images.
• To demonstrate that representation similarity, measured via RSA, correlates strongly with downstream transfer learning performance.
• To show that models trained on the same task may not generalize best due to domain shifts, and RSA can identify better alternatives.

Proposed method

• Uses Representation Similarity Analysis (RSA) to compute correlation-based similarity scores between deep neural network representations of pre-trained models.
• Computes similarity scores using only a few randomly selected images from each task, avoiding any fine-tuning or additional training.
• Applies RSA to compare representations of models trained on diverse visual tasks (e.g., classification, segmentation, depth estimation) from the Taskonomy dataset.
• Employs a small, lightweight model trained on Pascal VOC as a reference to compute similarity scores with Taskonomy models.
• Ranks models based on their RSA similarity to the reference model, enabling efficient model selection for transfer learning.
• Validates the method’s robustness across varying model sizes, number of images, and training stages in ablation studies.

Experimental results

Research questions

• RQ1 Can RSA be used to efficiently estimate the similarity between visual tasks without fine-tuning?
• RQ2 Does RSA similarity correlate with downstream transfer learning performance?
• RQ3 Is the similarity ranking robust to variations in model size, number of images, and training stage?
• RQ4 Can RSA identify better pre-trained models for transfer learning than those from the same task, especially under domain shift?
• RQ5 Does the method outperform existing approaches like Taskonomy in computational efficiency while maintaining predictive accuracy?

Key findings

• For 16 out of 17 Taskonomy tasks, the model selected by RSA was in the top-5 based on actual transfer learning performance.
• A strong correlation (ρ = 0.70, rs = 0.76) was observed between RSA similarity scores and transfer learning performance across 17 Taskonomy tasks.
• On Pascal VOC semantic segmentation, models with higher RSA similarity scores achieved better mIoU scores, with the top-performing model (scene class) achieving 0.6529 mIoU.
• The semantic segmentation model from Taskonomy had a lower RSA score than most 3D and semantic tasks and showed lower transfer performance, indicating domain shift effects.
• Models initialized with high-RSA-score tasks (e.g., object class, scene class) outperformed even ImageNet-pretrained models in some cases, with mIoU reaching 0.6529 compared to 0.6083 for random initialization.
• The method is robust to model size and training data quantity, as small models with few images produced similar similarity rankings to large, fully trained models.