[Paper Review] Virtual Mixup Training for Unsupervised Domain Adaptation

Xudong Mao, Yun Ma+3 more|arXiv (Cornell University)|May 10, 2019

Domain Adaptation and Few-Shot LearningComputer Science46 references49 citations

TL;DR

Introduces Virtual Mixup Training (VMT) to impose local Lipschitzness in-between training points for unsupervised domain adaptation, improving a VADA-based model across six benchmarks, notably MNIST to SVHN.

ABSTRACT

We study the problem of unsupervised domain adaptation which aims to adapt models trained on a labeled source domain to a completely unlabeled target domain. Recently, the cluster assumption has been applied to unsupervised domain adaptation and achieved strong performance. One critical factor in successful training of the cluster assumption is to impose the locally-Lipschitz constraint to the model. Existing methods only impose the locally-Lipschitz constraint around the training points while miss the other areas, such as the points in-between training data. In this paper, we address this issue by encouraging the model to behave linearly in-between training points. We propose a new regularization method called Virtual Mixup Training (VMT), which is able to incorporate the locally-Lipschitz constraint to the areas in-between training data. Unlike the traditional mixup model, our method constructs the combination samples without using the label information, allowing it to apply to unsupervised domain adaptation. The proposed method is generic and can be combined with most existing models such as the recent state-of-the-art model called VADA. Extensive experiments demonstrate that VMT significantly improves the performance of VADA on six domain adaptation benchmark datasets. For the challenging task of adapting MNIST to SVHN, VMT can improve the accuracy of VADA by over 30\%. Code is available at \url{https://github.com/xudonmao/VMT}.

Motivation & Objective

• Motivate and improve unsupervised domain adaptation by better enforcing the cluster assumption.
• Address the limitation of existing locally-Lipschitz regularizers that only cover training points.
• Introduce a mixup-based regularizer that extends Lipschitz constraints to in-between data regions.
• Enable applicability to unsupervised domain adaptation by using virtual labels in mixup.
• Demonstrate that VMT yields state-of-the-art results on standard domain adaptation benchmarks.

Proposed method

• Propose Virtual Mixup Training (VMT) to enforce linear changes between training samples and their virtual labels in-between data points.
• Replace real labels with virtual labels (predicted by the classifier) in the mixup formulation to suit unsupervised domain adaptation.
• Mix up on logits (pre-softmax layer) to improve stability, replacing mixup-on-probabilities.
• Combine VMT with existing models (notably VADA) in a joint objective that includes source/target losses, VAT, and conditional entropy.
• Use beta-distributed mixing coefficients to form convex combinations of samples and their virtual labels, then penalize KL divergence between f(tilde{x}) and tilde{y}.
• Optionally apply an iterative refinement stage (DIRT-T) on the target domain after VMT initialization to further optimize the cluster assumption.

Experimental results

Research questions

• RQ1 How can we extend the locally-Lipschitz constraint to the regions between training samples for unsupervised domain adaptation?
• RQ2 Does mixing on logits with virtual labels improve stability and performance compared to mixing on probabilities?
• RQ3 Can VMT consistently improve over state-of-the-art VADA across multiple domain adaptation benchmarks?
• RQ4 What is the impact of combining VMT with virtual adversarial training (VAT) and DIRT-T on target-domain clustering and alignment?

Key findings

• VMT improves VADA performance across six standard visual domain adaptation benchmarks.
• Mixing on logits yields more stable and higher accuracy than mixing on probabilities, especially on challenging MNIST→SVHN tasks.
• VMT achieves state-of-the-art results on several tasks and significantly narrows the gap to an upper-bound target-trained classifier (e.g., MNIST→SVHN without instance normalization).
• Combining VMT with DIRT-T further enhances target-domain clustering and alignment in many settings.
• VMT is computationally cheaper than VAT while providing complementary benefits when used together.