[Paper Review] Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation

Shiqi Yang, Yaxing Wang+3 more|arXiv (Cornell University)|Oct 8, 2021

Domain Adaptation and Few-Shot Learning58 references102 citations

TL;DR

This paper tackles source-free domain adaptation by leveraging the intrinsic local neighborhood structure of target features, using reciprocal and expanded neighborhoods with affinity weighting and self-regularization to align target predictions without access to source data.

ABSTRACT

Domain adaptation (DA) aims to alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g. due to data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain in the absence of source data. Our method is based on the observation that target data, which might no longer align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity. We observe that higher affinity should be assigned to reciprocal neighbors, and propose a self regularization loss to decrease the negative impact of noisy neighbors. Furthermore, to aggregate information with more context, we consider expanded neighborhoods with small affinity values. In the experimental results we verify that the inherent structure of the target features is an important source of information for domain adaptation. We demonstrate that this local structure can be efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of-the-art performance on several 2D image and 3D point cloud recognition datasets. Code is available in https://github.com/Albert0147/SFDA_neighbors.

Motivation & Objective

• Motivate SFDA by the practical constraint of not having access to source data during adaptation.
• Hypothesize that target features form meaningful clusters even under domain shift and can be exploited for labeling consistency.
• Leverage reciprocal nearest neighbors to assign higher supervision strength and reduce noisy neighbor impact.
• Incorporate expanded neighborhoods to capture broader local context while controlling noise via affinity weighting.
• Demonstrate state-of-the-art performance on multiple 2D image and 3D point cloud datasets.

Proposed method

• Define target feature affinity using reciprocal nearest neighbors (RNN) and non-reciprocal neighbors (nRNN).
• Use memory banks to store target features and their prediction scores for efficient neighbor retrieval.
• Formulate a neighborhood-based loss that encourages label agreement with neighbors weighted by affinity (L_N).
• Introduce a self-regularization term to mitigate the influence of noisy neighbors (L_self).
• Incorporate an expanded neighborhood (L_E) by considering neighbors of neighbors with a smaller affinity, capturing broader context.
• Combine affinity-weighted neighborhood supervision with a diversity (clustering) loss to prevent degenerate solutions (L_div).

Experimental results

Research questions

• RQ1 Can target data without source labels be effectively aligned by exploiting local neighborhood structure in the target feature space?
• RQ2 Do reciprocal nearest neighbors provide more reliable supervision than non-reciprocal neighbors in SFDA?
• RQ3 Does expanding the neighborhood (neighbors of neighbors) improve adaptation when tempered with affinity weights?
• RQ4 How do self-regularization and prediction diversity influence stability and performance of neighborhood-based SFDA?
• RQ5 What is the empirical performance of NRC on standard 2D and 3D SFDA benchmarks compared to prior methods?

Key findings

• Achieves competitive to state-of-the-art performance on multiple SFDA benchmarks, including Office-31, Office-Home, VisDA, and PointDA-10.
• Reciprocal nearest neighbors yield stronger supervision signals than non-reciprocal neighbors, improving clustering of target features.
• Expanded neighborhoods provide additional context and improve results when combined with affinity weighting; simply enlarging K without affinity can degrade performance.
• Self-regularization reduces the negative impact of noisy neighbors and stabilizes training.
• Ablation studies show the proposed NRC components (N, E, A, and L_div) collectively contribute to improved accuracy across datasets.