[Paper Review] VSE++: Improved Visual-Semantic Embeddings.

Fartash Faghri, David J. Fleet+2 more|arXiv (Cornell University)|Jul 18, 2017

Multimodal Machine Learning ApplicationsComputer Science143 citations

TL;DR

This paper proposes VSE++ to improve image-caption retrieval by replacing the original rank loss with a hard negative mining strategy that penalizes the most difficult negative examples. The method achieves state-of-the-art performance, improving R@1 by 21% on MS-COCO and more than doubling R@1 on Flickr30K compared to prior work.

ABSTRACT

This paper investigates the problem of image-caption retrieval using joint visual-semantic embeddings. We introduce a very simple change to the loss function used in the original formulation by Kiros et al. (2014), which leads to drastic improvements in the retrieval performance. In particular, the original paper uses the rank loss which computes the sum of violations across the negative training examples. Instead, we penalize the model according to the hardest negative examples. We then make several additional modifications according to the current best practices in image-caption retrieval. We showcase our model on the MS-COCO and Flickr30K datasets through comparisons and ablation studies. On MS-COCO, we improve caption retrieval by 21% in R@1 with respect to the original formulation. Our results outperform the state-of-the-art results by 8.8% in caption retrieval and 11.3% in image retrieval at R@1. On Flickr30K, we more than double R@1 as reported by Kiros et al. (2014) in both image and caption retrieval, and achieve near state-of-the-art performance. We further show that similar improvements also apply to the Order-embeddings by Vendrov et al. (2015) which builds on a similar loss function.

Motivation & Objective

• To improve image-caption retrieval performance using joint visual-semantic embeddings.
• To address the limitations of the original rank loss in VSE, which averages violations across all negative examples.
• To explore the impact of focusing on the hardest negative examples during training.
• To adapt current best practices in visual-semantic embedding to achieve state-of-the-art results.
• To demonstrate generalization of the proposed method to related models like Order-embeddings.

Proposed method

• Replace the original rank loss with a hard negative mining approach that penalizes only the most difficult negative example per positive pair.
• Apply standard improvements in deep learning for vision-language tasks, such as batch hard mining and normalization techniques.
• Use a Siamese network architecture to embed images and captions into a shared embedding space.
• Optimize the model using contrastive loss with hard negative mining to improve margin between positive and negative pairs.
• Integrate normalization and learning rate scheduling to stabilize training and improve convergence.
• Extend the method to Order-embeddings, showing its broader applicability.

Experimental results

Research questions

• RQ1 Does focusing on the hardest negative examples in the loss function lead to better retrieval performance than averaging over all negatives?
• RQ2 How does the proposed hard negative mining strategy compare to the original rank loss in terms of R@1 and R@5 metrics?
• RQ3 Can the improvements from hard negative mining be generalized to other models using similar loss functions, such as Order-embeddings?
• RQ4 To what extent do standard deep learning best practices enhance visual-semantic embedding performance on image-caption retrieval?
• RQ5 What is the performance gain of the proposed method on benchmark datasets like MS-COCO and Flickr30K?

Key findings

• On MS-COCO, VSE++ improves caption retrieval R@1 by 21% compared to the original VSE formulation.
• VSE++ achieves state-of-the-art performance on MS-COCO, outperforming prior SOTA by 8.8% in caption retrieval and 11.3% in image retrieval at R@1.
• On Flickr30K, VSE++ more than doubles the R@1 performance compared to the original VSE formulation reported by Kiros et al. (2014).
• The model achieves near state-of-the-art performance on Flickr30K despite the significant improvement over the baseline.
• The hard negative mining approach generalizes effectively to the Order-embeddings model, demonstrating broader applicability.
• The ablation studies confirm that the hard negative loss is the primary contributor to performance gains, with additional improvements from normalization and training practices.