[Paper Review] Producing augmentation-invariant embeddings from real-life imagery

Sergio Manuel Papadakis, Sanjay Addicam|arXiv (Cornell University)|Dec 6, 2021

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper presents a novel method for generating augmentation-invariant image embeddings using CNNs with ArcFace training, ensemble learning, and two query normalization techniques. By leveraging synthetic augmentations, multi-scale backbones, and a progressive training scheme, the approach achieved a 0.59 micro-Average Precision in the Facebook AI Image Similarity Challenge, securing 2nd place in Phase 2.

ABSTRACT

This article presents an efficient way to produce feature-rich, high-dimensionality embedding spaces from real-life images. The features produced are designed to be independent from augmentations used in real-life cases which appear on social media. Our approach uses convolutional neural networks (CNN) to produce an embedding space. An ArcFace head was used to train the model by employing automatically produced augmentations. Additionally, we present a way to make an ensemble out of different embeddings containing the same semantic information, a way to normalize the resulting embedding using an external dataset, and a novel way to perform quick training of these models with a high number of classes in the ArcFace head. Using this approach we achieved the 2nd place in the 2021 Facebook AI Image Similarity Challenge: Descriptor Track.

Motivation & Objective

• To develop a robust embedding space that remains invariant to real-world image augmentations commonly found in social media.
• To address class imbalance and limited reference set access in large-scale image similarity benchmarks.
• To improve model generalization by training on synthetic augmentations derived from real images.
• To normalize query embeddings using external training data to enhance similarity score consistency.
• To achieve state-of-the-art performance in the 2021 Facebook AI Image Similarity Challenge Descriptor Track.

Proposed method

• Trained multiple CNN backbones (EfficientNetV2, NFNet) with ArcFace heads using synthetic augmentations to increase training diversity.
• Employed Generalized Mean pooling (GeM) with trainable p parameter followed by a bias-free linear layer to produce 256D embeddings.
• Applied L2 normalization to final embeddings to ensure unit vector representation and improve similarity computation.
• Used a progressive training strategy with increasing class counts in the ArcFace head to accelerate convergence and improve generalization.
• Proposed two query normalization methods: (1) scaling query embeddings outward from the unit sphere using nearest training neighbors, and (2) moving embeddings along the mean direction away from the 100 closest training embeddings.
• Combined multiple models via ensemble averaging and applied external datasets (ImageNet, Deepfake Detection) to enrich training data, though the latter was later discarded due to redundancy and training instability.

Experimental results

Research questions

• RQ1 Can a deep learning model produce image embeddings that are invariant to diverse real-world image augmentations?
• RQ2 How can model generalization be improved under severe class imbalance and limited access to reference data?
• RQ3 What is the optimal way to normalize query embeddings to ensure similarity scores remain consistent across different augmentations?
• RQ4 Can ensemble learning and progressive training with large-scale ArcFace heads significantly improve performance in large-scale image similarity tasks?
• RQ5 Does using external datasets with facial content improve embedding quality for face-related image augmentations?

Key findings

• The proposed method achieved a micro-Average Precision (μAP) of 0.59 in Phase 2 of the Facebook AI Image Similarity Challenge, securing 2nd place.
• Method 2 for query normalization—moving embeddings along the mean direction away from the 100 closest training embeddings—improved μAP from 0.53 to 0.59 in Phase 2.
• Using a progressive training scheme with increasing ArcFace head classes enabled faster convergence and better generalization on high-accuracy tasks.
• The ensemble of multiple models with different backbones and input sizes outperformed individual models, demonstrating the effectiveness of model diversity.
• External datasets like ImageNet improved training data diversity, but the Deepfake Detection dataset was ultimately abandoned due to high redundancy and negative impact on training stability.
• The final model achieved high performance despite the prohibition of using the reference set during training, demonstrating strong generalization to unseen augmentations.