[Paper Review] Local Similarity-Aware Deep Feature Embedding

Chen Huang, Chen Change Loy+1 more|arXiv (Cornell University)|Oct 27, 2016

Domain Adaptation and Few-Shot LearningComputer Science28 references116 citations

TL;DR

Introduces a Position-Dependent Deep Metric (PDDM) that adapts similarity measurement to local feature structure, enabling online hard sample mining and joint metric-embedding learning for improved deep feature embeddings.

ABSTRACT

Existing deep embedding methods in vision tasks are capable of learning a compact Euclidean space from images, where Euclidean distances correspond to a similarity metric. To make learning more effective and efficient, hard sample mining is usually employed, with samples identified through computing the Euclidean feature distance. However, the global Euclidean distance cannot faithfully characterize the true feature similarity in a complex visual feature space, where the intraclass distance in a high-density region may be larger than the interclass distance in low-density regions. In this paper, we introduce a Position-Dependent Deep Metric (PDDM) unit, which is capable of learning a similarity metric adaptive to local feature structure. The metric can be used to select genuinely hard samples in a local neighborhood to guide the deep embedding learning in an online and robust manner. The new layer is appealing in that it is pluggable to any convolutional networks and is trained end-to-end. Our local similarity-aware feature embedding not only demonstrates faster convergence and boosted performance on two complex image retrieval datasets, its large margin nature also leads to superior generalization results under the large and open set scenarios of transfer learning and zero-shot learning on ImageNet 2010 and ImageNet-10K datasets.

Motivation & Objective

• Motivate the need for a local adaptive similarity metric instead of a global Euclidean metric in deep embedding.
• Propose a pluggable PDDM unit that learns a local metric using feature differences and absolute position.
• Jointly optimize the PDDM metric and embedding through a large-margin double-header hinge loss.
• Show that local similarity-aware embedding improves retrieval on fine-grained datasets and generalizes to transfer/zero-shot settings.

Proposed method

• Propose a Position-Dependent Deep Metric (PDDM) that uses feature difference u = |f(x_i)-f(x_j)| and feature mean v = (f(x_i)+f(x_j))/2 as inputs.
• Compute two separate transformed representations u′ and v′ through dedicated fully connected layers, followed by concatenation and a final scoring layer to produce S_i,j.
• Normalize features to unit hypersphere before processing to ensure comparability.
• Train PDDM end-to-end with the embedding CNN, sharing parameters across quadruplets and using four identical CNNs for the quadruplet features.
• Introduce a double-header hinge loss: a metric loss E_m that uses hard quadruplets within mini-batches to maximize separation between positive and negative similarity distributions, and an embedding loss E_e that enforces margin between hard positive and hard negative feature distances.
• Joint objective combines E_m, E_e, and L2 regularization on network parameters to jointly optimize metric and embedding.

Experimental results

Research questions

• RQ1 Can a locally adaptive similarity metric improve hard sample mining over global Euclidean or Mahalanobis metrics?
• RQ2 Does online PDDM-guided hard sample mining lead to faster convergence and better embedding quality for image retrieval?
• RQ3 Do jointly learned PDDM and embeddings generalize to transfer learning and zero-shot learning settings?
• RQ4 Is the combination of score-level and feature-level supervision beneficial for embedding quality?

Key findings

• PDDM yields more distinguishable similarity score distributions than the global Euclidean metric, enabling better hard sample mining.
• PDDM+Quadruplet and PDDM-based ranking achieve faster convergence and higher Recall@K on CUB-200-2011 and CARS196 compared to prior methods.
• The learned embeddings show large margins and robust clustering under Euclidean distance in challenging, heterogeneous feature spaces.
• PDDM and local similarity-aware embeddings improve transfer learning performance on ImageNet-10K and zero-shot learning on ImageNet-2010 compared to several baselines.
• Hard quadruplet mining via PDDM remains computationally efficient with near-linear batch complexity and avoids dense pairwise distance computation.