[Paper Review] Exploring Uncertainty in Conditional Multi-Modal Retrieval Systems

Ahmed Taha, Yi-Ting Chen+3 more|arXiv (Cornell University)|Jan 23, 2019

Human Pose and Action Recognition52 references4 citations

TL;DR

This paper reformulates triplet loss as a regression problem to enable Monte Carlo (MC) sampling and epistemic uncertainty estimation via dropout in conditional multi-modal retrieval. It proposes a multi-modal conditional retrieval network that disentangles embeddings for different similarity types, achieving state-of-the-art performance on person re-identification and a 6% improvement in uncertainty-rich autonomous driving scenarios using the Honda Driving Dataset (HDD).

ABSTRACT

We cast visual retrieval as a regression problem by posing triplet loss as a regression loss. This enables epistemic uncertainty estimation using dropout as a Bayesian approximation framework in retrieval. Accordingly, Monte Carlo (MC) sampling is leveraged to boost retrieval performance. Our approach is evaluated on two applications: person re-identification and autonomous car driving. Comparable state-of-the-art results are achieved on multiple datasets for the former application. We leverage the Honda driving dataset (HDD) for autonomous car driving application. It provides multiple modalities and similarity notions for ego-motion action understanding. Hence, we present a multi-modal conditional retrieval network. It disentangles embeddings into separate representations to encode different similarities. This form of joint learning eliminates the need to train multiple independent networks without any performance degradation. Quantitative evaluation highlights our approach competence, achieving 6% improvement in a highly uncertain environment.

Motivation & Objective

• To enable uncertainty estimation in retrieval systems by reformulating ranking losses as regression problems.
• To improve retrieval performance in high-uncertainty environments such as autonomous driving using Monte Carlo sampling.
• To develop a multi-modal conditional retrieval framework that jointly learns representations across different similarity notions (e.g., goal-oriented vs. stimulus-driven actions).
• To reduce computational cost and improve generalization by sharing representation learning across multiple similarity types instead of training separate networks.
• To evaluate the method on both person re-identification and autonomous navigation tasks, demonstrating robustness and performance gains in uncertain conditions.

Proposed method

• Reformulates triplet loss as a regression loss to enable Bayesian approximation via dropout, allowing epistemic uncertainty estimation.
• Applies Monte Carlo (MC) sampling during inference by performing multiple forward passes with dropout active, aggregating predictions to reduce uncertainty.
• Employs a conditional similarity network (CSN) to disentangle fused multi-modal embeddings (e.g., camera and CAN sensor data) into separate representations for distinct similarity types.
• Uses average fusion to combine modality-specific features before applying trainable masks for conditional retrieval per similarity type.
• Leverages a DropoutWrapper on RNN layers to maintain uncertainty estimation throughout sequence modeling in temporal retrieval tasks.
• Trains end-to-end using standard backpropagation with weight regularization, while enabling uncertainty quantification through MC dropout inference.

Experimental results

Research questions

• RQ1 Can triplet loss be effectively reformulated as a regression problem to enable uncertainty estimation in retrieval systems?
• RQ2 How does Monte Carlo sampling with dropout improve retrieval performance in high-uncertainty environments such as autonomous driving?
• RQ3 Can a single multi-modal conditional retrieval network jointly learn multiple similarity types (e.g., goal-oriented and stimulus-driven actions) without performance degradation?
• RQ4 To what extent does uncertainty estimation via MC dropout improve retrieval accuracy on ambiguous or noisy actions like 'stop for light' and 'stop for congestion'?
• RQ5 Can shared representation learning across similarity types reduce computational cost while maintaining or improving performance compared to independent models?

Key findings

• The proposed method achieves comparable state-of-the-art results on the Market-1501 and DukeMTMC-reID person re-identification datasets.
• On the Honda Driving Dataset (HDD), the method achieves a 6% improvement in retrieval performance in highly uncertain environments.
• For stimulus-driven actions, MC sampling with 100 samples increases macro mAP from 40.61 (baseline) to 45.13, with the largest gains seen in high-uncertainty actions like 'stop for congestion' (from 71.63 to 80.35 mAP).
• The top two retrieval results are robust to lighting variations and preserve environmental layout, indicating strong generalization and semantic consistency in learned embeddings.
• The method demonstrates superior performance on ambiguous actions such as 'stop for light' and 'stop for congestion', where visual cues are critical to disambiguate from similar actions.
• The use of MC sampling leads to significant performance gains—especially in high-uncertainty scenarios—without requiring additional model parameters or training, though at the cost of increased inference time.