[论文解读] Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts

Raymond A. Yeh, Jinjun Xiong+3 more|arXiv (Cornell University)|Mar 29, 2018

Multimodal Machine Learning Applications被引用 42

一句话总结

该论文将文本锚定视为对所有边界框的全局能量最小化，利用图像概念分数图和分支界定搜索实现精确推断，从而获得更丰富的候选项并提供可解释的词–图像概念嵌入；在提交时，它在 Flickr 30k Entities 和 ReferItGame 上超越了现有方法。

ABSTRACT

Textual grounding is an important but challenging task for human-computer interaction, robotics and knowledge mining. Existing algorithms generally formulate the task as selection from a set of bounding box proposals obtained from deep net based systems. In this work, we demonstrate that we can cast the problem of textual grounding into a unified framework that permits efficient search over all possible bounding boxes. Hence, the method is able to consider significantly more proposals and doesn't rely on a successful first stage hypothesizing bounding box proposals. Beyond, we demonstrate that the trained parameters of our model can be used as word-embeddings which capture spatial-image relationships and provide interpretability. Lastly, at the time of submission, our approach outperformed the current state-of-the-art methods on the Flickr 30k Entities and the ReferItGame dataset by 3.08% and 7.77% respectively.

研究动机与目标

• Motivate the textual grounding problem and its reliance on region proposals.
• Propose a unified, exact inference framework that searches over all bounding boxes using image concepts.
• Enable interpretability by exposing learned word–concept embeddings.
• Show empirical gains over state-of-the-art on Flickr 30k Entities and ReferItGame.

提出的方法

• Formulate grounding as E(x, y, w) = sum_{s in S} sum_{c in C} w_{s,c} φ_c(x, y, w_r).
• Represent image concepts as score maps (word priors, geometric cues, semantic segmentation, detections).
• Solve for the global minimizer ˆy = arg min_y E(x, y, w) with an efficient branch-and-bound algorithm (Alg. 1).
• Train parameters w using a structured SVM objective with IoU loss, via loss-augmented inference and a cutting-plane approach.
• Use integral images and precomputed caches to accelerate lower-bound bounds for branch-and-bound.

实验结果

研究问题

• RQ1 Can textual grounding be solved by exact optimization over a large bounding-box space rather than a small proposal set?
• RQ2 Do image-concept score maps enable accurate, interpretable grounding and robust performance across datasets?
• RQ3 Can learned weights w_{s,c} serve as meaningful word embeddings capturing spatial-image relations?
• RQ4 What are the empirical gains over existing grounding methods on Flickr 30k Entities and ReferItGame?
• RQ5 Is the proposed method computationally efficient enough for practical use?

主要发现

• Achieves state-of-the-art accuracy on Flickr 30k Entities (Table 1: 51.63% with Prior+Geo+Seg+Det and 53.97% with Prior+Geo+Seg+bDet).
• Achieves state-of-the-art accuracy on ReferItGame (Table 2: 34.70% with Prior+Geo+Seg+Det).
• Demonstrates significant improvements over baselines such as SCRC, DSPE, GroundeR, and CCA across datasets (Table 1 results vs. 2016–2017 methods).
• Word–concept weights ws,c learned by the model function as interpretable word embeddings, capturing spatial-image relationships (Fig. 6).
• Inference runs with branch-and-bound provide global optimality and fast runtime comparable to or faster than competing methods (section on computational efficiency).