[Paper Review] Detecting Visual Relationships with Deep Relational Networks

Bo Dai, Yuqi Zhang+1 more|arXiv (Cornell University)|Apr 11, 2017

Advanced Image and Video Retrieval Techniques50 references68 citations

TL;DR

The paper proposes Deep Relational Network (DR-Net) to jointly infer triplets (subject, predicate, object) for visual relationship detection by modeling spatial configurations and statistical dependencies within a neural network, outperforming prior methods on VRD and sVG datasets.

ABSTRACT

Relationships among objects play a crucial role in image understanding. Despite the great success of deep learning techniques in recognizing individual objects, reasoning about the relationships among objects remains a challenging task. Previous methods often treat this as a classification problem, considering each type of relationship (e.g. "ride") or each distinct visual phrase (e.g. "person-ride-horse") as a category. Such approaches are faced with significant difficulties caused by the high diversity of visual appearance for each kind of relationships or the large number of distinct visual phrases. We propose an integrated framework to tackle this problem. At the heart of this framework is the Deep Relational Network, a novel formulation designed specifically for exploiting the statistical dependencies between objects and their relationships. On two large datasets, the proposed method achieves substantial improvement over state-of-the-art.

Motivation & Objective

• Motivate the need for robust visual relationship detection beyond independent object recognition.
• Introduce a framework that predicts (subject, predicate, object) triplets by leveraging spatial and statistical relations.
• Develop DR-Net to unroll probabilistic inference into a deep network for end-to-end learning.
• Show that combining appearance, spatial masks, and statistical dependencies yields superior performance.

Proposed method

• Detect objects with Faster R-CNN and extract appearance features for each candidate object.
• Form object pairs and apply a pair filter to discard unlikely relations.
• For retained pairs, extract an appearance feature from an enclosing box and two dual spatial masks (32x32) for subject and object, compressed to 64D.
• Propose Deep Relational Network (DR-Net) to perform iterative posterior updates for s, r, o by unrolling inference into neural layers (q_s, q_r, q_o) and sharing weights across inference units.
• Model relations with symmetric weight constraints between components and enable end-to-end discriminative training.
• Compare DR-Net to CRF-based formulations and other baselines, showing DR-Net provides significant gains in predicate recognition.

Experimental results

Research questions

• RQ1 Can a jointly learned model capture the statistical dependencies between subject, predicate, and object to improve visual relationship detection?
• RQ2 Does integrating appearance, spatial configurations, and learned relational inferences outperform classification-based or CRF-based approaches?
• RQ3 How does the number of inference units and weight sharing affect DR-Net performance?
• RQ4 Is the framework effective for scene graph generation beyond relation detection?

Key findings

• DR-Net achieves substantial recall gains over prior methods on VRD and sVG for predicate recognition (e.g., DR-Net recall@50/100 significantly higher than baselines).
• Incorporating both appearance and spatial configuration improves performance beyond either component alone.
• DR-Net's ability to exploit statistical dependencies (vs. CRFs) yields meaningful perplexity reductions and better predicate disambiguation.
• More inference units generally improve recall, with diminishing returns depending on weight sharing, and non-shared configurations can achieve higher gains with enough units.
• The framework extends to scene graph generation, producing higher similarity to ground truth graphs when using DR-Net configurations.