[Paper Review] Attention-based Ensemble for Deep Metric Learning

Wonsik Kim, Bhavya Goyal+3 more|arXiv (Cornell University)|Apr 2, 2018

Advanced Image and Video Retrieval Techniques34 references22 citations

TL;DR

This paper proposes an attention-based ensemble (ABE-M) for deep metric learning that enhances model diversity by training multiple learners with distinct attention masks, each focusing on different image regions. By introducing a divergence loss to further encourage dissimilarity in attended features, ABE-M achieves state-of-the-art performance on CARS-196, CUB-200-2011, SOP, and in-shop clothes retrieval benchmarks with fewer parameters than baseline ensembles.

ABSTRACT

Deep metric learning aims to learn an embedding function, modeled as deep neural network. This embedding function usually puts semantically similar images close while dissimilar images far from each other in the learned embedding space. Recently, ensemble has been applied to deep metric learning to yield state-of-the-art results. As one important aspect of ensemble, the learners should be diverse in their feature embeddings. To this end, we propose an attention-based ensemble, which uses multiple attention masks, so that each learner can attend to different parts of the object. We also propose a divergence loss, which encourages diversity among the learners. The proposed method is applied to the standard benchmarks of deep metric learning and experimental results show that it outperforms the state-of-the-art methods by a significant margin on image retrieval tasks.

Motivation & Objective

• To improve deep metric learning performance through model ensembling with enhanced feature diversity.
• To address the lack of architectural design for inducing diversity in feature embeddings across ensemble members.
• To develop a parameter-efficient ensemble framework that maintains high performance with reduced model size.
• To validate that attention-based attention masks and a divergence loss can jointly improve both individual and ensemble model performance.

Proposed method

• The method employs an M-way attention-based ensemble (ABE-M) where a shared backbone network is used with M separate attention modules, each learning to focus on different spatial regions of the input image.
• Each attention module generates a unique feature representation by attending to different parts of the object, promoting diversity in the learned embeddings.
• A divergence loss is introduced to explicitly minimize similarity between the attention maps of different learners, encouraging them to focus on distinct image regions.
• The framework is trained end-to-end with standard metric learning losses (e.g., triplet or contrastive loss) combined with the proposed divergence loss.
• The architecture shares early convolutional layers across all learners, reducing parameter count while maintaining high performance.
• The method is evaluated on standard benchmarks including CARS-196, CUB-200-2011, SOP, and in-shop clothes retrieval.

Experimental results

Research questions

• RQ1 Can attention-based feature selection across multiple learners improve performance in deep metric learning?
• RQ2 Does explicitly encouraging diversity in attention maps lead to better generalization and higher retrieval accuracy?
• RQ3 Can a shared-backbone ensemble with attention modules outperform standard M-head ensembles with fewer parameters?
• RQ4 Does the proposed divergence loss improve both individual learner performance and ensemble accuracy?

Key findings

• ABE-8 with 512-dimensional features achieved 98.7% Recall@50 on the in-shop clothes retrieval dataset, outperforming the previous SOTA A-BIER by 0.3%.
• On CUB-200-2011 (cropped), ABE-8 achieved 92.2% Recall@30, surpassing A-BIER (92.1%) and Margin (90.6%).
• On SOP, ABE-8 reached 98.2% Recall@1000, exceeding A-BIER (97.8%) and Margin (98.0%).
• ABE-8 on CARS-196 (cropped) achieved 95.5% Recall@8, outperforming A-BIER (95.6%) and HDC (93.8%).
• The proposed ABE-4 and ABE-8 models achieved higher performance than the M-heads baseline while using fewer parameters.
• The divergence loss improved both individual model performance and ensemble accuracy, demonstrating its effectiveness in promoting feature diversity.