[Paper Review] Graph Matching with Bi-level Noisy Correspondence

Yijie Lin, Mouxing Yang+4 more|arXiv (Cornell University)|Dec 8, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper introduces Bi-level Noisy Correspondence (BNC), a novel challenge in graph matching arising from noisy node and edge-level correspondences due to annotation errors. The authors propose COMMON, a contrastive matching method with momentum distillation that uses a robust quadratic contrastive loss and adaptive noise penalization via a momentum teacher model, achieving state-of-the-art performance on three real-world datasets despite noisy annotations.

ABSTRACT

In this paper, we study a novel and widely existing problem in graph matching (GM), namely, Bi-level Noisy Correspondence (BNC), which refers to node-level noisy correspondence (NNC) and edge-level noisy correspondence (ENC). In brief, on the one hand, due to the poor recognizability and viewpoint differences between images, it is inevitable to inaccurately annotate some keypoints with offset and confusion, leading to the mismatch between two associated nodes, i.e., NNC. On the other hand, the noisy node-to-node correspondence will further contaminate the edge-to-edge correspondence, thus leading to ENC. For the BNC challenge, we propose a novel method termed Contrastive Matching with Momentum Distillation. Specifically, the proposed method is with a robust quadratic contrastive loss which enjoys the following merits: i) better exploring the node-to-node and edge-to-edge correlations through a GM customized quadratic contrastive learning paradigm; ii) adaptively penalizing the noisy assignments based on the confidence estimated by the momentum teacher. Extensive experiments on three real-world datasets show the robustness of our model compared with 12 competitive baselines. The code is available at https://github.com/XLearning-SCU/2023-ICCV-COMMON.

Motivation & Objective

• To address the under-explored problem of Bi-level Noisy Correspondence (BNC) in graph matching, where both node-level and edge-level correspondences are corrupted by annotation errors.
• To develop a method that is robust to noisy correspondences without requiring pre-identified noisy labels or verification data.
• To integrate both linear (node-to-node) and quadratic (edge-to-edge) geometric information into a contrastive learning framework for improved alignment.
• To leverage the memorization behavior of deep networks by using a momentum teacher model to generate high-quality pseudo-targets for noise-aware training.

Proposed method

• The method employs a robust quadratic contrastive loss that explicitly models both node-to-node and edge-to-edge affinities using a graph-geometric consistency regularizer.
• It introduces two novel regularizers to embed quadratic structural information into the contrastive learning objective, enhancing correlation modeling between nodes and edges.
• A momentum teacher network is used to generate stable pseudo-targets during training, leveraging the fact that deep networks first memorize clean patterns before fitting noise.
• The model adaptively penalizes low-confidence assignments by comparing student predictions with the momentum teacher's outputs, reducing the impact of noisy correspondences.
• The architecture consists of a VGG16-based image encoder, SplineCNN for graph convolution with Delaunay triangulation, and a two-layer projection head for feature projection to 256-dim embeddings.
• Final node similarity and edge adjacency matrices are computed via dot product of projected features, forming the basis for the contrastive loss.

Experimental results

Research questions

• RQ1 How does the presence of bi-level noisy correspondence—noisy node and edge correspondences—impact the performance of existing graph matching methods?
• RQ2 Can a contrastive learning framework be effectively extended to model both linear and quadratic affinities in graph matching under noisy conditions?
• RQ3 To what extent can a momentum distillation strategy improve robustness to noisy correspondences in graph matching without requiring pre-labeled noise information?
• RQ4 How does the integration of geometric consistency regularizers enhance the model's ability to learn reliable node and edge-level correspondences?

Key findings

• The proposed COMMON method achieves state-of-the-art performance on three real-world graph matching benchmarks, outperforming 12 competitive baselines.
• Extensive ablation studies confirm that the quadratic contrastive loss and momentum distillation are both essential for robustness to noisy correspondences.
• The method significantly improves matching accuracy on challenging cases with low recognizability and large viewpoint differences, as shown in visualizations on Pascal VOC and SPair-71k.
• The momentum distillation strategy effectively reduces the influence of noisy pairs by leveraging the teacher model’s stable predictions, even when the student model is exposed to corrupted annotations.
• Despite the addition of a momentum network, inference speed remains identical to standard models, with training time only ~1.4× that of a base model.