[Paper Review] Contextual Similarity Aggregation with Self-attention for Visual Re-ranking

Jianbo Ouyang, Hui Wu+3 more|arXiv (Cornell University)|Oct 26, 2021

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes a self-attention-based visual re-ranking method that improves instance image retrieval by aggregating contextual similarity among top-ranked images. It constructs affinity features via similarity to top-L anchors, refines them using a transformer encoder to capture inter-image context, and re-ranks candidates using refined features—achieving state-of-the-art performance with low computational overhead across four benchmarks.

ABSTRACT

In content-based image retrieval, the first-round retrieval result by simple visual feature comparison may be unsatisfactory, which can be refined by visual re-ranking techniques. In image retrieval, it is observed that the contextual similarity among the top-ranked images is an important clue to distinguish the semantic relevance. Inspired by this observation, in this paper, we propose a visual re-ranking method by contextual similarity aggregation with self-attention. In our approach, for each image in the top-K ranking list, we represent it into an affinity feature vector by comparing it with a set of anchor images. Then, the affinity features of the top-K images are refined by aggregating the contextual information with a transformer encoder. Finally, the affinity features are used to recalculate the similarity scores between the query and the top-K images for re-ranking of the latter. To further improve the robustness of our re-ranking model and enhance the performance of our method, a new data augmentation scheme is designed. Since our re-ranking model is not directly involved with the visual feature used in the initial retrieval, it is ready to be applied to retrieval result lists obtained from various retrieval algorithms. We conduct comprehensive experiments on four benchmark datasets to demonstrate the generality and effectiveness of our proposed visual re-ranking method.

Motivation & Objective

• To improve instance image retrieval by leveraging contextual similarity among top-ranked images, which is often ignored in pairwise similarity-based re-ranking.
• To develop a re-ranking method that is independent of the original visual feature extraction pipeline, enabling compatibility with diverse retrieval algorithms.
• To enhance model robustness and generalization through a novel data augmentation scheme for the re-ranking task.
• To achieve high performance with low computational cost by avoiding database-wide operations and scaling only with the top-K retrieval list.

Proposed method

• Affinity features are computed for each of the top-K images by measuring their similarity to a set of L anchor images selected from the top-L results.
• A transformer encoder dynamically refines the affinity features by attending to other top-K candidates, modeling contextual relationships based on similarity between affinity features.
• The model is trained with a contrastive loss to increase similarity between relevant images and an MSE loss to preserve original affinity features, preventing overfitting.
• A new data augmentation strategy is introduced to improve model generalization and reduce overfitting during training.
• During inference, refined affinity features are used to recompute cosine similarity scores between query and top-K candidates for final ranking.
• The method is agnostic to the initial retrieval feature, allowing plug-and-play integration with any retrieval pipeline.

Experimental results

Research questions

• RQ1 Can modeling contextual similarity among top-ranked images improve visual re-ranking performance beyond pairwise similarity?
• RQ2 How can self-attention mechanisms effectively aggregate contextual information from multiple top-ranked images to refine re-ranking scores?
• RQ3 Can a re-ranking model be designed to be independent of the initial feature extraction method while maintaining high performance?
• RQ4 What impact does a novel data augmentation scheme have on the robustness and generalization of the re-ranking model?
• RQ5 How does the proposed method compare in efficiency and scalability to existing re-ranking approaches, especially on large-scale databases?

Key findings

• The proposed method achieves state-of-the-art performance on four benchmark datasets, outperforming existing re-ranking methods including LAttQE, DFS, and GSS in most settings.
• The method achieves higher performance than LAttQE, a leading query expansion-based method, demonstrating the effectiveness of contextual similarity aggregation.
• When combined with DFS or GSS, the proposed re-ranking method further improves retrieval performance, indicating strong compatibility and complementarity.
• The model with data augmentation achieves higher performance than the baseline without it, confirming the effectiveness of the proposed augmentation strategy.
• The method has lower time and memory complexity than state-of-the-art methods like DFS and GSS, with latency independent of database size and only scaling with top-K list size.