[Paper Review] Latent Domain Learning with Dynamic Residual Adapters

Lucas Deecke, Timothy M. Hospedales+1 more|arXiv (Cornell University)|Jun 1, 2020

Domain Adaptation and Few-Shot Learning36 references4 citations

TL;DR

This paper proposes Dynamic Residual Adapters (DRA), a method for latent domain learning that enables deep networks to adaptively distribute feature corrections across multiple unseen domains without requiring domain annotations. By using a learnable gating mechanism that dynamically routes residual updates based on input content, DRA achieves robust performance across small and large domains, outperforming both standard models and domain-supervised baselines in multi-domain image classification tasks.

ABSTRACT

A practical shortcoming of deep neural networks is their specialization to a single task and domain. While recent techniques in domain adaptation and multi-domain learning enable the learning of more domain-agnostic features, their success relies on the presence of domain labels, typically requiring manual annotation and careful curation of datasets. Here we focus on a less explored, but more realistic case: learning from data from multiple domains, without access to domain annotations. In this scenario, standard model training leads to the overfitting of large domains, while disregarding smaller ones. We address this limitation via dynamic residual adapters, an adaptive gating mechanism that helps account for latent domains, coupled with an augmentation strategy inspired by recent style transfer techniques. Our proposed approach is examined on image classification tasks containing multiple latent domains, and we showcase its ability to obtain robust performance across these. Dynamic residual adapters significantly outperform off-the-shelf networks with much larger capacity, and can be incorporated seamlessly with existing architectures in an end-to-end manner.

Motivation & Objective

• To address the challenge of overfitting to large domains in multi-domain learning when domain annotations are unavailable.
• To enable deep networks to learn robust, generalizable features across multiple latent domains without relying on manually curated domain labels.
• To develop an efficient, plug-and-play module that can be seamlessly integrated into existing architectures for end-to-end training.

Proposed method

• Introduces Dynamic Residual Adapters (DRA), a learnable gating mechanism that dynamically assigns residual corrections to different domains based on input features.
• Uses a differentiable routing mechanism that allows each layer to adaptively activate multiple residual paths depending on input content.
• Employs a residual adapter module with learnable parameters $ h_{\alpha_k} $, where the gate $ g_k $ determines the contribution of each path.
• Applies an augmentation strategy inspired by style transfer to improve generalization and domain robustness.
• Trains the entire model end-to-end, enabling joint optimization of feature extraction and domain-aware routing.
• Visualizes activation paths across layers to show that semantically similar domains (e.g., photo and art painting) activate similar gate patterns.

Experimental results

Research questions

• RQ1 Can a deep network learn to generalize across multiple latent domains without access to domain annotations?
• RQ2 How can a model avoid overfitting to large domains while maintaining performance on smaller, underrepresented ones?
• RQ3 Can a dynamic, adaptive routing mechanism outperform fixed, domain-supervised routing in multi-domain learning?
• RQ4 To what extent do learned gate pathways reflect semantic similarity between domains?
• RQ5 Does the proposed method surpass domain-supervised approaches even when domain labels are available?

Key findings

• The DRA model achieves a weighted average accuracy of 87.01% on the Visual Decathlon benchmark, outperforming a standard joint model by 2.69 percentage points.
• On the PACS dataset, DRA surpasses domain-supervised models even when the latter use carefully curated domain labels.
• The model reduces performance drop on small domains: for example, Aircraft accuracy improves by 20.59% compared to the joint model.
• Visualization of activation paths shows that semantically similar domains (e.g., photo and art painting) cluster together in the gate activation space.
• The method maintains high performance on large domains while significantly improving performance on small ones, demonstrating effective domain balancing.
• The use of Gumbel-softmax to enforce discrete gate assignments leads to performance degradation, confirming that soft, dynamic routing is more effective than hard clustering.