[Paper Review] Rethinking Diversified and Discriminative Proposal Generation for Visual Grounding

Yu Zhou, Jun Yu+4 more|arXiv (Cornell University)|May 9, 2018

Multimodal Machine Learning Applications3 citations

TL;DR

This paper proposes Diversified and Discriminative Proposal Networks (DDPN), a novel approach that jointly optimizes for diversity and discrimination in object proposal generation for visual grounding. By generating high-quality proposals and training a strong baseline with KLD and regression losses, DDPN achieves state-of-the-art performance, including an 18.8% improvement on ReferItGame and 8.2% on Flickr30k Entities over prior methods.

ABSTRACT

Visual grounding aims to localize an object in an image referred to by a textual query phrase. Various visual grounding approaches have been proposed, and the problem can be modularized into a general framework: proposal generation, multi-modal feature representation, and proposal ranking. Of these three modules, most existing approaches focus on the latter two, with the importance of proposal generation generally neglected. In this paper, we rethink the problem of what properties make a good proposal generator. We introduce the diversity and discrimination simultaneously when generating proposals, and in doing so propose Diversified and Discriminative Proposal Networks model (DDPN). Based on the proposals generated by DDPN, we propose a high performance baseline model for visual grounding and evaluate it on four benchmark datasets. Experimental results demonstrate that our model delivers significant improvements on all the tested data-sets (e.g., 18.8\% improvement on ReferItGame and 8.2\% improvement on Flickr30k Entities over the existing state-of-the-arts respectively)

Motivation & Objective

• To address the underexplored role of proposal generation in visual grounding, which is often treated as a fixed module.
• To investigate what makes a high-quality proposal generator for visual grounding, focusing on diversity and discrimination.
• To design a unified framework that simultaneously enhances proposal diversity and discriminative power.
• To establish a strong baseline for visual grounding using DDPN-generated proposals and novel training losses.
• To validate the effectiveness of DDPN across multiple benchmark datasets through extensive ablation and comparison.

Proposed method

• Propose Diversified and Discriminative Proposal Networks (DDPN), a two-stream network that jointly optimizes for proposal diversity and discriminative quality.
• Use a diversity loss based on feature clustering to encourage proposals to cover distinct object regions.
• Apply a discrimination loss using contrastive learning to enhance the separability of proposals from each other and from background.
• Integrate DDPN with a simple feature concatenation module for multimodal fusion between visual and textual features.
• Train the full model with two novel losses: Kullback-Leibler (KLD) divergence loss with soft labels to model proposal context, and smoothed L1 regression loss for bounding box refinement.
• Use VGG-16 and ResNet-101 as backbones to evaluate the impact of feature representation capacity on performance.

Experimental results

Research questions

• RQ1 What properties make a proposal generator effective for visual grounding, and how can diversity and discrimination be jointly optimized?
• RQ2 Can a unified proposal generation framework significantly improve visual grounding performance compared to standard RPNs or selective search?
• RQ3 How do KLD loss with soft labels and regression loss contribute to improved localization accuracy?
• RQ4 To what extent does the choice of backbone network (e.g., VGG-16 vs. ResNet-101) affect the performance of the DDPN-based system?
• RQ5 How does the proposed DDPN-based baseline compare to state-of-the-art methods across multiple benchmarks?

Key findings

• The DDPN-based model achieves a 18.8% absolute improvement on ReferItGame (63.0% vs. 44.2% for prior SOTA) and an 8.2% improvement on Flickr30k Entities (73.3% vs. 65.1%).
• Using ResNet-101 as the backbone yields a 3–4 point performance gain over VGG-16, demonstrating the importance of feature representation capacity.
• The ablation study confirms that both diversity and discrimination components in DDPN are essential, with ablation on either component leading to significant performance drops.
• The KLD loss with soft labels improves contextual modeling of proposals, contributing to better ranking performance.
• The smoothed L1 regression loss effectively refines bounding box predictions, reducing localization errors and improving IoU scores.
• The proposed baseline model outperforms all prior state-of-the-art methods on all four benchmark datasets: RefCOCO, RefCOCO+, Flickr30k Entities, and ReferItGame.