[Paper Review] Bag of Tricks and A Strong baseline for Image Copy Detection

Wenhao Wang, Weipu Zhang+2 more|arXiv (Cornell University)|Nov 13, 2021

Advanced Image and Video Retrieval Techniques34 references4 citations

TL;DR

This paper presents a strong, unsupervised pre-training-based baseline with a novel descriptor stretching strategy for image copy detection, achieving third place among 526 teams in the NeurIPS 2021 Facebook AI Image Similarity Challenge Descriptor Track. The method improves retrieval accuracy by stabilizing inter-query score variance through descriptor stretching and leverages self-supervised learning (Barlow-Twins) and data augmentation to enhance robustness in real-world social media scenarios with massive distractor images.

ABSTRACT

Image copy detection is of great importance in real-life social media. In this paper, a bag of tricks and a strong baseline are proposed for image copy detection. Unsupervised pre-training substitutes the commonly-used supervised one. Beyond that, we design a descriptor stretching strategy to stabilize the scores of different queries. Experiments demonstrate that the proposed method is effective. The proposed baseline ranks third out of 526 participants on the Facebook AI Image Similarity Challenge: Descriptor Track. The code and trained models are available at https://github.com/WangWenhao0716/ISC-Track2-Submission.

Motivation & Objective

• To address the challenge of image copy detection in real-world social media environments with billions of distractor images and diverse image transformations.
• To improve retrieval performance by replacing supervised pre-training with unsupervised self-supervised pre-training, particularly using Barlow-Twins.
• To stabilize inter-query matching score variance in the descriptor space, which is critical for consistent performance under the Descriptor Track's evaluation protocol.
• To develop a practical, effective baseline that outperforms existing methods on the challenging ISC2021 benchmark.

Proposed method

• Utilizes Barlow-Twins self-supervised pre-training on ImageNet instead of supervised pre-training, which empirically yields superior performance.
• Employs a deep metric learning baseline enhanced with a bag of tricks, including GeM pooling, WaveBlock, a high-dimensional projector, learnable matrix, triplet loss with hard mining, and cross-entropy loss.
• Applies diverse data augmentations—including random cropping, color jitter, blurring, rotation, and overlay operations (text, emoji, images)—to improve representation robustness.
• Introduces descriptor stretching during inference to rescale descriptors and stabilize scores across queries, approximating score normalization without violating the Descriptor Track's constraints.
• Uses multi-scale inference (200×200 to 400×400) with L2-normalized feature fusion to enhance robustness and accuracy.
• Employs YOLOv5 for overlay detection to identify and handle image overlays, improving performance by ~1% on matching pairs.

Experimental results

Research questions

• RQ1 Can unsupervised self-supervised pre-training outperform supervised pre-training in image copy detection under real-world conditions?
• RQ2 How can descriptor score variance across queries be stabilized to improve retrieval consistency in the absence of score normalization?
• RQ3 To what extent do data augmentation and multi-scale inference enhance robustness to unseen image transformations in copy detection?
• RQ4 What is the impact of overlay detection and descriptor stretching on retrieval performance in a needle-in-haystack setting?

Key findings

• The proposed baseline achieved a micro-average precision of 0.73017 on the ISC2021 test set, ranking third out of 526 participants in the Descriptor Track.
• Unsupervised pre-training with Barlow-Twins improved performance by over 14% compared to supervised pre-training, increasing micro-average precision from 0.39089 to 0.53218.
• Descriptor stretching alone improved micro-average precision by 17.26%, from 0.70481 to 0.73017, with no improvement in per-query metrics, indicating score stabilization as the key mechanism.
• Overlay detection via YOLOv5 contributed approximately 1% improvement in performance by identifying and handling image overlays.
• Multi-scale testing improved performance from 71.49% to 73.02% micro-average precision, confirming its effectiveness in robust feature extraction.
• The method demonstrated strong generalization, achieving high performance on a large-scale benchmark with diverse transformations, including cropping, blurring, color jitter, and image overlays.