[Paper Review] Minimal-Entropy Correlation Alignment for Unsupervised Deep Domain Adaptation

Pietro Morerio, Jacopo Cavazza+1 more|arXiv (Cornell University)|Nov 28, 2017

Domain Adaptation and Few-Shot Learning84 citations

TL;DR

MECA ties covariance (second-order) alignment on SPD manifolds with entropy minimization, using a geodesic distance and an entropy-based unsupervised hyperparameter selector to improve unsupervised deep domain adaptation.

ABSTRACT

In this work, we face the problem of unsupervised domain adaptation with a novel deep learning approach which leverages on our finding that entropy minimization is induced by the optimal alignment of second order statistics between source and target domains. We formally demonstrate this hypothesis and, aiming at achieving an optimal alignment in practical cases, we adopt a more principled strategy which, differently from the current Euclidean approaches, deploys alignment along geodesics. Our pipeline can be implemented by adding to the standard classification loss (on the labeled source domain), a source-to-target regularizer that is weighted in an unsupervised and data-driven fashion. We provide extensive experiments to assess the superiority of our framework on standard domain and modality adaptation benchmarks.

Motivation & Objective

• Motivate unsupervised domain adaptation where target labels are unavailable.
• Show that optimal correlation alignment minimizes source cross-entropy and target entropy.
• Propose a geodesic (SPD manifold) distance for covariance alignment over Euclidean metrics.
• Introduce an entropy-driven criterion to select the regularization strength without target labels.
• Demonstrate state-of-the-art performance on standard domain adaptation benchmarks.

Proposed method

• Formulate correlation alignment as aligning second-order statistics via covariance matrices on an SPD manifold.
• Use a squared log-Euclidean geodesic distance between source and target covariances as the alignment loss.
• Combine supervised source cross-entropy with the geodesic alignment loss in a single MECA objective.
• Propose an entropy-based, data-driven method to choose the regularization weight without target labels.
• Prove a theoretical link showing optimal correlation alignment implies minimum target entropy (Theorem 1).
• Provide gradient-based training compatible with backpropagation through standard deep nets.

Experimental results

Research questions

• RQ1 Does optimal covariance alignment implicitly minimize the target domain entropy in unsupervised domain adaptation?
• RQ2 Does a geodesic SPD-distance yield better alignment and targets than Euclidean penalties for covariance alignment?
• RQ3 Can entropy-based criteria effectively select hyperparameters without target labels?
• RQ4 How does MECA perform against state-of-the-art unsupervised domain adaptation methods on standard benchmarks?

Key findings

• Optimal correlation alignment minimizes the sum of source cross-entropy and target entropy (Theorem 1).
• Geodesic (log-Euclidean) SPD distance outperforms Euclidean penalties for covariance alignment in practice.
• Entropy on the target can be used to drive hyperparameter selection without target labels, via an unsupervised criterion.
• MECA achieves competitive or superior results compared with DSN, DTN, GRL, ADDA, TRIPLE, and Deep CORAL on standard benchmarks.
• MECA yields notable improvements over Deep CORAL on SVHN→MNIST and NYUD, and a strong performance on SYN→SVHN.
• Across reported tasks, MECA (E + C) outperforms several entropy-related and alignment baselines.