[Paper Review] The Devil is in the Middle: Exploiting Mid-level Representations for Cross-Domain Instance Matching

Qian Yu, Xiaobin Chang+3 more|arXiv (Cornell University)|Nov 22, 2017

Advanced Image and Video Retrieval Techniques37 references80 citations

TL;DR

The paper shows that mid-level CNN feature maps contain discriminative, domain-invariant information for cross-domain instance matching and should be fused with final-layer features, achieving new state-of-the-art on FG-SBIR and person ReID.

ABSTRACT

Many vision problems require matching images of object instances across different domains. These include fine-grained sketch-based image retrieval (FG-SBIR) and Person Re-identification (person ReID). Existing approaches attempt to learn a joint embedding space where images from different domains can be directly compared. In most cases, this space is defined by the output of the final layer of a deep neural network (DNN), which primarily contains features of a high semantic level. In this paper, we argue that both high and mid-level features are relevant for cross-domain instance matching (CDIM). Importantly, mid-level features already exist in earlier layers of the DNN. They just need to be extracted, represented, and fused properly with the final layer. Based on this simple but powerful idea, we propose a unified framework for CDIM. Instantiating our framework for FG-SBIR and ReID, we show that our simple models can easily beat the state-of-the-art models, which are often equipped with much more elaborate architectures.

Motivation & Objective

• Motivate and address cross-domain instance matching (CDIM) where objects are captured in different domains (e.g., photo-sketch, multi-camera ReID).
• Argue that mid-level features exist in early CNN layers and are crucial for CDIM alongside high-level features.
• Propose a unified architecture pattern to fuse mid-level representations with final-layer features.
• Demonstrate the approach with FG-SBIR and person ReID instantiations and achieve state-of-the-art results.

Proposed method

• Propose a three-part CDIM design pattern: a CNN base network, a fusion module, and a loss module.
• Extract mid-level feature maps from chosen middle layers, transform them into a mid-level feature vector, and fuse with the final-layer features without excessive dimensionality reduction.
• For FG-SBIR, flatten mid-level conv5 features to preserve spatial information and concatenate with final fc7 features, trained with a triplet ranking loss.
• For ReID, apply global average pooling to mid-level res5a/res5b features, concatenate with pooled top-layer features, and train with a softmax classification loss.
• Demonstrate that mid-level features require deep supervision and task-specific pooling strategies (flattening for FG-SBIR, GAP for ReID).
• Show that fusing mid-level and high-level features yields superior performance over vanilla final-layer representations and attention-based baselines.

Experimental results

Research questions

• RQ1 Do mid-level representations in CNNs contain domain-invariant discriminative cues useful for CDIM?
• RQ2 Can a simple fusion of mid-level feature maps with final-layer features outperform more elaborate attention-based CDIM methods?
• RQ3 What pooling strategy and layer choices maximize the effectiveness of mid-level feature fusion for FG-SBIR and person ReID?

Key findings

• Mid-layer feature maps carry discriminative, spatially localized cues useful for cross-domain matching.
• A simple fusion of mid-level and final-layer features with deep supervision improves CDIM performance over baselines.
• The proposed approach achieves new state-of-the-art results on FG-SBIR datasets (shoes, chairs, handbags) and three person ReID benchmarks (Market-1501, DukeMTMC_reID, CUHK03-New).
• Task-specific pooling (flattening for FG-SBIR; GAP for ReID) and fusing mid-level layers near the final layer are beneficial.
• Using mid-level features with different base networks (Sketch-a-Net, ResNet-50, InceptionV3) consistently improves performance, validating the framework’s applicability.