[Paper Review] Exploring Uncertainty Measures for Image-Caption Embedding-and-Retrieval Task

Kenta Hama, Takashi Matsubara+2 more|arXiv (Cornell University)|Apr 9, 2019

Multimodal Machine Learning Applications54 references4 citations

TL;DR

This paper proposes two uncertainty measures—feature uncertainty (via regression-based model averaging) and posterior uncertainty (via classification-based reliability scoring)—for image-caption retrieval systems. It demonstrates that posterior uncertainty consistently improves retrieval performance and reliability assessment across datasets, while feature uncertainty offers limited reliability for out-of-distribution or ambiguous queries.

ABSTRACT

With the wide development of black-box machine learning algorithms, particularly deep neural network (DNN), the practical demand for the reliability assessment is rapidly rising. On the basis of the concept that `Bayesian deep learning knows what it does not know,' the uncertainty of DNN outputs has been investigated as a reliability measure for the classification and regression tasks. However, in the image-caption retrieval task, well-known samples are not always easy-to-retrieve samples. This study investigates two aspects of image-caption embedding-and-retrieval systems. On one hand, we quantify feature uncertainty by considering image-caption embedding as a regression task, and use it for model averaging, which can improve the retrieval performance. On the other hand, we further quantify posterior uncertainty by considering the retrieval as a classification task, and use it as a reliability measure, which can greatly improve the retrieval performance by rejecting uncertain queries. The consistent performance of two uncertainty measures is observed with different datasets (MS COCO and Flickr30k), different deep learning architectures (dropout and batch normalization), and different similarity functions.

Motivation & Objective

• To address the lack of reliability assessment in image-caption embedding-and-retrieval systems, especially under dataset shift or ambiguous queries.
• To investigate whether uncertainty quantification from Bayesian deep learning can improve retrieval performance and detect unreliable predictions.
• To compare the effectiveness of two uncertainty measures—feature uncertainty (regression view) and posterior uncertainty (classification view)—in diverse settings.
• To evaluate the robustness of these uncertainty measures across different datasets (MS COCO, Flickr30k, RecipeQA), architectures (VGG19, VSE++), and similarity functions.

Proposed method

• Treat image-caption embedding as a regression task to estimate feature uncertainty via Monte Carlo dropout, using model averaging over multiple forward passes to reduce variance.
• Reframe the retrieval task as a classification problem to compute posterior uncertainty as the entropy of the predicted similarity distribution across candidate captions.
• Apply temperature scaling to the posterior uncertainty to improve calibration and reliability estimation.
• Train and evaluate the VSE++ model with VGG19 on MS COCO, then test on Flickr30k and RecipeQA to assess generalization and dataset shift robustness.
• Use rank-based loss (e.g., contrastive loss) during training but estimate uncertainty via posterior distributions during inference.
• Evaluate uncertainty measures using retrieval metrics (e.g., R@1, R@5, R@10) and compare performance with and without uncertainty-based filtering or model averaging.

Experimental results

Research questions

• RQ1 Can feature uncertainty derived from embedding regression improve retrieval performance through model averaging?
• RQ2 Can posterior uncertainty derived from classification-based similarity scoring serve as a reliable reliability measure for image-caption retrieval?
• RQ3 How do these uncertainty measures perform across different datasets, especially under covariate shift (e.g., MS COCO to Flickr30k or RecipeQA)?
• RQ4 Why does feature uncertainty fail to detect out-of-distribution or ambiguous queries despite high variance in embeddings?
• RQ5 How do architectural choices (e.g., dropout, batch normalization) and similarity functions affect the reliability of uncertainty estimates?

Key findings

• Posterior uncertainty significantly improves retrieval performance by filtering out unreliable queries, especially in ambiguous or out-of-distribution cases.
• Feature uncertainty improves retrieval accuracy via model averaging across multiple Monte Carlo forward passes, but does not reliably indicate prediction confidence.
• Posterior uncertainty is consistently higher on the Flickr30k and especially RecipeQA datasets compared to MS COCO, indicating its sensitivity to dataset shift and distributional shift.
• Feature uncertainty shows similar distributions across MS COCO, Flickr30k, and RecipeQA, despite significant performance degradation on Flickr30k, indicating poor reliability for out-of-domain detection.
• The qualitative analysis reveals that posterior uncertainty captures confusion among similar targets (e.g., baseball swings), while feature uncertainty reflects local image complexity rather than semantic ambiguity.
• The posterior uncertainty measure is robust across different deep learning architectures (e.g., VGG19 with VSE++) and similarity functions, demonstrating generalizability and strong reliability assessment capability.