[Paper Review] VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

Fartash Faghri, David J. Fleet+2 more|arXiv (Cornell University)|Jul 18, 2017

Multimodal Machine Learning Applications29 references579 citations

TL;DR

VSE++ introduces a hard negative mining–inspired max-margin hinge loss (MH) for visual-semantic embedding training, yielding large gains in image-caption retrieval on MS-COCO and Flickr30K, especially with stronger image encoders and data augmentation.

ABSTRACT

We present a new technique for learning visual-semantic embeddings for cross-modal retrieval. Inspired by hard negative mining, the use of hard negatives in structured prediction, and ranking loss functions, we introduce a simple change to common loss functions used for multi-modal embeddings. That, combined with fine-tuning and use of augmented data, yields significant gains in retrieval performance. We showcase our approach, VSE++, on MS-COCO and Flickr30K datasets, using ablation studies and comparisons with existing methods. On MS-COCO our approach outperforms state-of-the-art methods by 8.8% in caption retrieval and 11.3% in image retrieval (at R@1).

Motivation & Objective

• Motivate learning robust visual-semantic embeddings for cross-modal retrieval (caption ↔ image).
• Investigate the effect of hard negative sampling within a joint embedding loss.
• Improve retrieval performance via a new loss (MH) and data augmentation/fine-tuning.
• Demonstrate gains with stronger image encoders (e.g., ResNet) and dataset augmentation.
• Provide ablations and comparisons to state-of-the-art baselines on MS-COCO and Flickr30K.

Proposed method

• Define a joint embedding with image and caption projections into a shared space and L2-normalize both embeddings.
• Replace the standard triplet/hyperplane hinge loss with a Max of Hinges (MH) loss that focuses on hardest negatives within mini-batches: ell(i,c)=max_c' [alpha+s(i,c')-s(i,c)]+max_i' [alpha+s(i',c)-s(i,c)].
• Compute negatives within each mini-batch (no extra mining cost) to emphasize hard negatives near positives.
• Optionally fine-tune the image encoder and augment data (RC, 10C, rV, etc.) for improved performance.
• Experiment with VGG19 and ResNet encoders, GRU-based caption encoder, and joint dimension D=1024.
• Evaluate using recall at K (R@K) for caption and image retrieval on MS-COCO and Flickr30K.

Experimental results

Research questions

• RQ1 Does incorporating hard negatives via MH loss improve R@1/5/10 and median rank for caption and image retrieval compared to baseline SH loss?
• RQ2 How do data augmentation and image encoder improvements (e.g., ResNet, fine-tuning) interact with MH loss to boost performance?
• RQ3 Is the MH loss robust to batch size and label noise due to intra-batch hard negatives?
• RQ4 Can MH loss improve other embedding approaches beyond VSE, e.g., order-embeddings?
• RQ5 What is the effect of training data size and augmentation on VSE++ performance?

Key findings

• On MS-COCO, VSE++ with ResNet152 and fine-tuning achieves R@1 of 64.6% for caption retrieval and 52.0% for image retrieval on 1K test images (Table 1/1.11).
• VSE++ consistently outperforms VSE0 and prior state-of-the-art methods, with an 8.8% absolute gain in caption R@1 and 11.3% in image R@1 over the best prior result on MS-COCO.
• Using harder negatives within mini-batches (MH loss) yields substantial gains beyond simply using a stronger image encoder (e.g., ResNet) or data augmentation alone (e.g., RC/ rV).
• The improvement persists across datasets (MS-COCO and Flickr30K) and training variants (1C, RC, rV; FT).
• MH loss benefits extending to other embedding approaches (Order++) when swapping SH with MH, achieving notable R@1 gains on MS-COCO.
• MH loss shows a warm-up period during training but overtakes SH after a few epochs; curriculum strategies offered but MH alone performed strong.