[Paper Review] Joint Pixel and Feature-level Domain Adaptation in the Wild.

Luan Tran, Kihyuk Sohn+3 more|arXiv (Cornell University)|Feb 28, 2018

Domain Adaptation and Few-Shot Learning11 references3 citations

TL;DR

This paper proposes a joint pixel and feature-level domain adaptation framework that combines domain adversarial entropy minimization at the feature level with 3D geometry-aware image synthesis and attribute-conditioned CycleGAN at the pixel level. The method achieves state-of-the-art performance in car recognition on unlabeled surveillance images by explicitly modeling named factors of variation (e.g., pose, lighting) and implicitly handling unknown factors through complementary adaptation strategies.

ABSTRACT

Recent developments in deep domain adaptation have allowed knowledge transfer from a labeled source domain to an unlabeled target domain at the level of intermediate features or input pixels. We propose that advantages may be derived by combining them, in the form of different insights that lead to a novel design and complementary properties that result in better performance. At the feature level, inspired by insights from semi-supervised learning in a domain adversarial neural network, we propose a novel regularization in the form of domain adversarial entropy minimization. Next, we posit that insights from computer vision are more amenable to injection at the pixel level and specifically address the key challenge of adaptation across different semantic levels. In particular, we use 3D geometry and image synthesization based on a generalized appearance flow to preserve identity across higher-level pose transformations, while using an attribute-conditioned CycleGAN to translate a single source into multiple target images that differ in lower-level properties such as lighting. We validate on a novel problem of car recognition in unlabeled surveillance images using labeled images from the web, handling explicitly specified, nameable factors of variation through pixel-level and implicit, unspecified factors through feature-level adaptation. Extensive experiments achieve state-of-the-art results, demonstrating the effectiveness of complementing feature and pixel-level information via our proposed domain adaptation method.

Motivation & Objective

• To improve domain adaptation in real-world scenarios where labeled source data (e.g., web images) differs significantly from unlabeled target data (e.g., surveillance footage).
• To address the challenge of adapting across both explicit, nameable factors of variation (e.g., pose, lighting) and implicit, unspecified factors through complementary adaptation strategies.
• To develop a unified framework that leverages both pixel-level and feature-level adaptation for improved generalization in cross-domain recognition.
• To validate the effectiveness of combining geometric image synthesis with domain adversarial learning in a real-world recognition task.

Proposed method

• Introduces domain adversarial entropy minimization as a novel regularization technique at the feature level, inspired by semi-supervised learning and domain adversarial networks.
• Uses 3D geometry and generalized appearance flow to synthesize identity-preserving images under varying poses, enabling robust adaptation across semantic-level transformations.
• Employs an attribute-conditioned CycleGAN to generate diverse target-domain images with variations in low-level properties such as lighting and color.
• Combines the outputs of pixel-level and feature-level adaptation modules to jointly optimize domain alignment and feature disentanglement.
• Leverages both explicit control over named factors (e.g., pose, lighting) and implicit modeling of unknown factors through adversarial training.
• Trains the model end-to-end using labeled source images and unlabeled target images, with no need for paired data between domains.

Experimental results

Research questions

• RQ1 Can combining pixel-level and feature-level domain adaptation improve performance in real-world, unconstrained recognition tasks?
• RQ2 How effectively can 3D geometry and appearance flow preserve identity during pose and viewpoint changes in image synthesis for domain adaptation?
• RQ3 To what extent does domain adversarial entropy minimization enhance feature-level domain alignment in the absence of explicit supervision?
• RQ4 Can attribute-conditioned image translation effectively model low-level domain shifts such as lighting and color variations?
• RQ5 How does the joint use of explicit and implicit adaptation strategies compare to single-level approaches in cross-domain car recognition?

Key findings

• The proposed method achieves state-of-the-art performance on a novel car recognition benchmark using web-sourced labeled images and unlabeled surveillance footage.
• Joint adaptation at both pixel and feature levels significantly outperforms methods relying solely on pixel-level or feature-level adaptation.
• The use of 3D geometry and appearance flow enables effective preservation of identity across large pose variations in synthesized target images.
• Attribute-conditioned CycleGAN successfully generates diverse, realistic target-domain images with controlled variations in lighting and color.
• Domain adversarial entropy minimization effectively reduces domain shift in intermediate features, improving generalization to unseen target domains.
• The method demonstrates strong robustness to both named (e.g., pose, lighting) and unnamed factors of variation in real-world surveillance settings.