[論文レビュー] DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning

Chi Zhang, Yujun Cai+2 more|arXiv (Cornell University)|Mar 15, 2020

Domain Adaptation and Few-Shot Learning参考文献 85被引用数 38

ひとこと要約

この論文は、局所的な画像領域間の微分可能な Earth Mover’s Distance (EMD) を用いて few-shot 画像分類を最適マッチング問題として formalize し、クロスリファレンス重み付け方式と k-shot タスク用の構造化全結合層を導入し、標準ベンチマークで最先端の結果を達成する。

ABSTRACT

In this work, we develop methods for few-shot image classification from a new perspective of optimal matching between image regions. We employ the Earth Mover's Distance (EMD) as a metric to compute a structural distance between dense image representations to determine image relevance. The EMD generates the optimal matching flows between structural elements that have the minimum matching cost, which is used to calculate the image distance for classification. To generate the important weights of elements in the EMD formulation, we design a cross-reference mechanism, which can effectively alleviate the adverse impact caused by the cluttered background and large intra-class appearance variations. To implement k-shot classification, we propose to learn a structured fully connected layer that can directly classify dense image representations with the EMD. Based on the implicit function theorem, the EMD can be inserted as a layer into the network for end-to-end training. Our extensive experiments validate the effectiveness of our algorithm which outperforms state-of-the-art methods by a significant margin on five widely used few-shot classification benchmarks, namely, miniImageNet, tieredImageNet, Fewshot-CIFAR100 (FC100), Caltech-UCSD Birds-200-2011 (CUB), and CIFAR-FewShot (CIFAR-FS). We also demonstrate the effectiveness of our method on the image retrieval task in our experiments.

研究の動機と目的

• Motivate few-shot classification as structured matching between local image regions rather than global embeddings.
• Develop a differentiable EMD layer that can be embedded into neural networks for end-to-end training.
• Propose a cross-reference mechanism to weight local regions to reduce background noise and enhance foreground relevance.
• Introduce a structured fully connected layer to enable k-shot classification using EMD-based distance to class prototypes.

提案手法

• Represent images as sets of local region embeddings extracted by FCN, grids, or random patches.
• Compute distance between two images via Earth Mover’s Distance with costs c_ij = 1 - (u_i^T v_j) / (||u_i|| ||v_j||).
• Generate node weights s_i and d_j with a cross-reference mechanism that compares region features across the two images.
• Embed the EMD optimization as a differentiable layer using KKT conditions and the implicit function theorem for end-to-end training.
• For k-shot, replace standard FC with a structured fully connected layer that classifies based on EMD between query features and class prototype regions.
• Provide a training protocol combining a pre-training step and episodic meta-training, plus iterative refinement of the structured FC layer.

実験結果

リサーチクエスチョン

• RQ1 Can a differentiable Earth Mover’s Distance between local image regions improve few-shot classification performance?
• RQ2 Does a cross-reference mechanism for weighting region contributions mitigate background clutter and intra-class variation?
• RQ3 Can a structured fully connected layer effectively perform k-shot classification using EMD-based distances?

主な発見

モデル	埋め込み	指標	5-way	10-way
ProtoNet	global	Euclidean	60.37	-
MatchingNet	global	cosine	63.08	47.09
FC	global	dot	59.41	44.08
FC	global	cosine	55.43	40.42
KNN	local	cosine	62.52	47.08
Prediction Fusion	local	cosine	62.38	47.04
DeepEMD-FCN	local	EMD	65.91	49.66

• DeepEMD-FCN with EMD outperforms baseline methods on 1-shot and 5-shot tasks across five benchmarks.
• 1-shot results show DeepEMD-FCN achieving 65.91 (5-way) and 49.66 (10-way) versus baselines like ProtoNet, MatchingNet, and FC variants.
• EMD with cross-reference weighting yields the best performance among EMD variants.
• The approach also improves image retrieval performance beyond classification tasks.
• The model supports end-to-end training by differentiating through the LP-based EMD layer.