[Paper Review] Deep CORAL: Correlation Alignment for Deep Domain Adaptation

Baochen Sun, Kate Saenko|arXiv (Cornell University)|Jul 6, 2016

Domain Adaptation and Few-Shot Learning19 references368 citations

TL;DR

Deep CORAL integrates a differentiable CORAL loss into deep networks to align source and target feature covariances, enabling end-to-end unsupervised domain adaptation with state-of-the-art results.

ABSTRACT

Deep neural networks are able to learn powerful representations from large quantities of labeled input data, however they cannot always generalize well across changes in input distributions. Domain adaptation algorithms have been proposed to compensate for the degradation in performance due to domain shift. In this paper, we address the case when the target domain is unlabeled, requiring unsupervised adaptation. CORAL is a "frustratingly easy" unsupervised domain adaptation method that aligns the second-order statistics of the source and target distributions with a linear transformation. Here, we extend CORAL to learn a nonlinear transformation that aligns correlations of layer activations in deep neural networks (Deep CORAL). Experiments on standard benchmark datasets show state-of-the-art performance.

Motivation & Objective

• Motivate unsupervised domain adaptation where the target domain is unlabeled.
• Extend CORAL to a deep, differentiable loss for end-to-end training.
• Learn nonlinear transformations that align correlations of layer activations across domains.
• Demonstrate superior performance on standard domain adaptation benchmarks.

Proposed method

• Define CORAL loss as the squared Frobenius norm between source and target covariance matrices.
• Compute covariances from batch activations at a chosen layer in the network.
• Integrate CORAL loss with standard classification loss in an end-to-end objective.
• Train with shared parameters across source and target pathways, using labeled source data and unlabeled target data.
• Apply CORAL loss to different layers or architectures; demonstrate on last classifier layer (fc8) of AlexNet.

Experimental results

Research questions

• RQ1 Can a differentiable CORAL loss be integrated into deep networks to achieve unsupervised domain adaptation?
• RQ2 Does aligning second-order statistics (covariances) of deep features improve target-domain performance without labeled target data?
• RQ3 How does Deep CORAL compare to existing unsupervised domain adaptation methods on standard benchmarks?

Key findings

• Deep CORAL achieves state-of-the-art performance on the Office benchmark compared to seven baselines.
• In 3 of 6 domain shifts, Deep CORAL obtains the highest accuracy; in the other 3, gains are modest (≤0.7).
• Combining classification loss with CORAL loss yields balanced improvements on source and target domains.
• Visual analyses show CORAL encourages smaller domain discrepancy during fine-tuning and preserves discriminative source performance.
• End-to-end CORAL loss integrates seamlessly with CNNs and can be applied to various layers and architectures.