[Paper Review] Visual Translation Embedding Network for Visual Relation Detection

Hanwang Zhang, Zawlin Kyaw+2 more|arXiv (Cornell University)|Feb 27, 2017

Multimodal Machine Learning ApplicationsComputer Science15 references61 citations

TL;DR

VTransE end-to-end network learns visual relations by embedding subject, predicate, and object into a low-dimensional relation space using translation embeddings, enabling simultaneous object detection and relation prediction with knowledge transfer between objects and relations.

ABSTRACT

Visual relations, such as "person ride bike" and "bike next to car", offer a comprehensive scene understanding of an image, and have already shown their great utility in connecting computer vision and natural language. However, due to the challenging combinatorial complexity of modeling subject-predicate-object relation triplets, very little work has been done to localize and predict visual relations. Inspired by the recent advances in relational representation learning of knowledge bases and convolutional object detection networks, we propose a Visual Translation Embedding network (VTransE) for visual relation detection. VTransE places objects in a low-dimensional relation space where a relation can be modeled as a simple vector translation, i.e., subject + predicate $\approx$ object. We propose a novel feature extraction layer that enables object-relation knowledge transfer in a fully-convolutional fashion that supports training and inference in a single forward/backward pass. To the best of our knowledge, VTransE is the first end-to-end relation detection network. We demonstrate the effectiveness of VTransE over other state-of-the-art methods on two large-scale datasets: Visual Relationship and Visual Genome. Note that even though VTransE is a purely visual model, it is still competitive to the Lu's multi-modal model with language priors.

Motivation & Objective

• Motivate the need for explicit visual relation modeling to bridge vision and language beyond object detection and captioning
• Propose a translation-embedding based representation for visual relations to handle long-tailed predicate distributions
• Introduce a differentiable feature extraction layer to enable knowledge transfer between objects and predicates in an end-to-end CNN
• Demonstrate that VTransE improves relation detection on VRD and Visual Genome datasets without language priors
• Show that end-to-end training enhances object detection via relational context

Proposed method

• Represent relations as s + p ≈ o in a low-dimensional relation space, with projection matrices Ws, Wo mapping subjects/objects to this space
• Use a translation vector tp for each predicate p to model visual relations, enabling a simple distance-based or softmax-based loss
• Introduce a differentiable Feature Extraction Layer that combines classeme, location, and bilinear-interpolated visual features into x_s and x_o, enabling end-to-end knowledge transfer
• Replace RoI pooling with bilinear interpolation to create smooth gradients for end-to-end training
• Train with a multi-task loss L = Lobj + 0.4 Lrel, where Lrel uses softmax over predicates and a translation-based scoring term
• Object detection uses Faster-RCNN with VGG-16, but with a bilinear interpolation based feature extraction to support relation learning
• Score for a relation combines object detection scores and predicate prediction scores as S_s,p,o = S_s + S_p + S_o

Experimental results

Research questions

• RQ1 Does embedding visual relations in a low-dimensional translation space improve predicate prediction compared to JointBox-style models?
• RQ2 Which features (classeme, location, visual) contribute most to relation detection and how does feature fusion affect different relation types?
• RQ3 Can end-to-end VTransE improve object detection through contextual relation learning and vice versa?
• RQ4 How does VTransE perform against state-of-the-art visual relation models on VRD and Visual Genome, including zero-shot scenarios?

Key findings

• VTransE outperforms JointBox on predicate prediction in VRD and VG datasets (R@50 and R@100)
• Adding all three feature types (classeme, location, visual) with learned scaling yields best relation detection across verbs, spatial, prepositions, and comparatives
• End-to-end training with VTransE improves object detection mAP on VRD and VG, demonstrating reciprocal learning between objects and relations
• VTransE (purely visual) outperforms Lu’s-VLK on several tasks, illustrating the benefit of translation embeddings for visual relations
• Two-stage VTransE (VTransE-2stage) lags behind end-to-end VTransE, underscoring the value of joint optimization
• Zero-shot results show substantial degradation across all methods, highlighting challenges in generalizing verb/object compositions without language priors