[Paper Review] A constrained clustering based approach for matching a collection of feature sets

Junchi Yan, Zhe Ren+2 more|arXiv (Cornell University)|Jun 12, 2016

Advanced Image and Video Retrieval Techniques3 references3 citations

TL;DR

This paper proposes a constrained clustering-based method for matching multiple feature sets using only unary node features, iteratively assigning features to clusters formed by other sets while updating centers. It achieves state-of-the-art performance, especially under noise and outliers, by leveraging cross-set consistency and global information synergistically.

ABSTRACT

In this paper, we consider the problem of finding the feature correspondences among a collection of feature sets, by using their point-wise unary features. This is a fundamental problem in computer vision and pattern recognition, which also closely relates to other areas such as operational research. Different from two-set matching which can be transformed to a quadratic assignment programming task that is known NP-hard, inclusion of merely unary attributes leads to a linear assignment problem for matching two feature sets. This problem has been well studied and there are effective polynomial global optimum solvers such as the Hungarian method. However, it becomes ill-posed when the unary attributes are (heavily) corrupted. The global optimal correspondence concerning the best score defined by the attribute affinity/cost between the two sets can be distinct to the ground truth correspondence since the score function is biased by noises. To combat this issue, we devise a method for matching a collection of feature sets by synergetically exploring the information across the sets. In general, our method can be perceived from a (constrained) clustering perspective: in each iteration, it assigns the features of one set to the clusters formed by the rest of feature sets, and updates the cluster centers in turn. Results on both synthetic data and real images suggest the efficacy of our method against state-of-the-arts.

Motivation & Objective

• Address the challenge of matching multiple feature sets when unary features are corrupted by noise, which degrades performance of standard pairwise methods.
• Overcome the limitations of pairwise linear assignment (e.g., Hungarian method) that fail to exploit global consistency across multiple sets.
• Develop a method that enforces correspondence consistency across all feature sets by jointly exploring information from all sets.
• Design an iterative clustering framework that alternately assigns features to clusters and updates cluster centers to improve matching accuracy.

Proposed method

• Formulate the multi-set feature matching problem as a constrained clustering task, where features from one set are assigned to clusters formed by the remaining sets.
• In each iteration, assign features of a selected set to clusters derived from the union of all other sets, using a nearest-neighbor strategy based on unary feature similarity.
• Update cluster centers iteratively by computing the centroid of assigned features, refining cluster representations across iterations.
• Use a consistency-based criterion to dynamically select the base feature set and update order, improving robustness to noisy or heterogeneous data.
• Introduce two variants: matchCluster (adaptive selection of base set and order) and matchCluster_fast (random selection), to evaluate sensitivity to initialization.
• Leverage the fact that unary-only matching is a linear assignment problem per pair, but extend it to multiple sets via clustering to exploit global structure.

Experimental results

Research questions

• RQ1 Can global information across multiple feature sets be effectively leveraged to improve matching accuracy when unary features are corrupted?
• RQ2 How does the proposed constrained clustering framework compare to pairwise Hungarian methods and other state-of-the-art multi-set matching approaches under noise and outliers?
• RQ3 Does adaptive selection of the base feature set and update order improve performance compared to random or fixed ordering?
• RQ4 To what extent does the method preserve correspondence consistency across multiple feature sets in real-world image sequences?

Key findings

• On synthetic data with increasing noise and outliers, the proposed method (matchCluster) outperforms all baselines, especially under high deformation and outlier rates.
• In real-world image sequences (CMU Hotel and Willow-ObjectClass), matchCluster performs competitively with state-of-the-art methods, even outperforming matchOpt in some settings.
• The adaptive version (matchCluster) significantly outperforms the fast variant (matchCluster_fast), particularly on real images, indicating that initialization quality matters in heterogeneous data.
• The Hungarian method applied pairwise (Hung_lin) performs worst due to lack of cross-set information, confirming the benefit of joint global optimization.
• Visual results show that matchCluster produces fewer incorrect correspondences (red lines) and more correct ones (green lines) than pairwise Hungarian matching.
• The method is robust to noise and outliers, with accuracy decreasing gradually under increasing disturbance, unlike baseline methods that degrade sharply.