[Paper Review] DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning

Chi Zhang, Yujun Cai+2 more|arXiv (Cornell University)|Mar 15, 2020

Domain Adaptation and Few-Shot Learning85 references38 citations

TL;DR

The paper formalizes few-shot image classification as an optimal matching problem using differentiable Earth Mover’s Distance (EMD) between local image regions, introduces a cross-reference weighting scheme, and a structured fully connected layer for k-shot tasks, achieving state-of-the-art results on standard benchmarks.

ABSTRACT

In this work, we develop methods for few-shot image classification from a new perspective of optimal matching between image regions. We employ the Earth Mover's Distance (EMD) as a metric to compute a structural distance between dense image representations to determine image relevance. The EMD generates the optimal matching flows between structural elements that have the minimum matching cost, which is used to calculate the image distance for classification. To generate the important weights of elements in the EMD formulation, we design a cross-reference mechanism, which can effectively alleviate the adverse impact caused by the cluttered background and large intra-class appearance variations. To implement k-shot classification, we propose to learn a structured fully connected layer that can directly classify dense image representations with the EMD. Based on the implicit function theorem, the EMD can be inserted as a layer into the network for end-to-end training. Our extensive experiments validate the effectiveness of our algorithm which outperforms state-of-the-art methods by a significant margin on five widely used few-shot classification benchmarks, namely, miniImageNet, tieredImageNet, Fewshot-CIFAR100 (FC100), Caltech-UCSD Birds-200-2011 (CUB), and CIFAR-FewShot (CIFAR-FS). We also demonstrate the effectiveness of our method on the image retrieval task in our experiments.

Motivation & Objective

• Motivate few-shot classification as structured matching between local image regions rather than global embeddings.
• Develop a differentiable EMD layer that can be embedded into neural networks for end-to-end training.
• Propose a cross-reference mechanism to weight local regions to reduce background noise and enhance foreground relevance.
• Introduce a structured fully connected layer to enable k-shot classification using EMD-based distance to class prototypes.

Proposed method

• Represent images as sets of local region embeddings extracted by FCN, grids, or random patches.
• Compute distance between two images via Earth Mover’s Distance with costs c_ij = 1 - (u_i^T v_j) / (||u_i|| ||v_j||).
• Generate node weights s_i and d_j with a cross-reference mechanism that compares region features across the two images.
• Embed the EMD optimization as a differentiable layer using KKT conditions and the implicit function theorem for end-to-end training.
• For k-shot, replace standard FC with a structured fully connected layer that classifies based on EMD between query features and class prototype regions.
• Provide a training protocol combining a pre-training step and episodic meta-training, plus iterative refinement of the structured FC layer.

Experimental results

Research questions

• RQ1 Can a differentiable Earth Mover’s Distance between local image regions improve few-shot classification performance?
• RQ2 Does a cross-reference mechanism for weighting region contributions mitigate background clutter and intra-class variation?
• RQ3 Can a structured fully connected layer effectively perform k-shot classification using EMD-based distances?

Key findings

• DeepEMD-FCN with EMD outperforms baseline methods on 1-shot and 5-shot tasks across five benchmarks.
• 1-shot results show DeepEMD-FCN achieving 65.91 (5-way) and 49.66 (10-way) versus baselines like ProtoNet, MatchingNet, and FC variants.
• EMD with cross-reference weighting yields the best performance among EMD variants.
• The approach also improves image retrieval performance beyond classification tasks.
• The model supports end-to-end training by differentiating through the LP-based EMD layer.