[Paper Review] Knowledge-Embedded Routing Network for Scene Graph Generation

Tianshui Chen, Weihao Yu+2 more|arXiv (Cornell University)|Mar 8, 2019

Multimodal Machine Learning Applications33 references45 citations

TL;DR

The paper introduces Knowledge-Embedded Routing Network (KERN) that explicitly encodes statistical object-pair and relationship knowledge in a structured graph to regularize scene graph predictions, yielding stronger performance than prior methods, especially for rare relationships on Visual Genome.

ABSTRACT

To understand a scene in depth not only involves locating/recognizing individual objects, but also requires to infer the relationships and interactions among them. However, since the distribution of real-world relationships is seriously unbalanced, existing methods perform quite poorly for the less frequent relationships. In this work, we find that the statistical correlations between object pairs and their relationships can effectively regularize semantic space and make prediction less ambiguous, and thus well address the unbalanced distribution issue. To achieve this, we incorporate these statistical correlations into deep neural networks to facilitate scene graph generation by developing a Knowledge-Embedded Routing Network. More specifically, we show that the statistical correlations between objects appearing in images and their relationships, can be explicitly represented by a structured knowledge graph, and a routing mechanism is learned to propagate messages through the graph to explore their interactions. Extensive experiments on the large-scale Visual Genome dataset demonstrate the superiority of the proposed method over current state-of-the-art competitors.

Motivation & Objective

• Address the long-tail problem in scene graph generation by leveraging prior statistical knowledge of object co-occurrences and their relationships.
• Represent statistical correlations as structured graphs and integrate them into graph neural networks to regulate the semantic space of relationship prediction.
• Model the interplay between objects and relationships with dedicated routing networks to improve prediction accuracy across SGGen, SGCls, and PredCls tasks.
• Evaluate on the Visual Genome dataset and compare with state-of-the-art methods, highlighting gains in mean recall (mR@K) and recall (R@K).

Proposed method

• Detect object regions with Faster R-CNN to obtain candidate boxes and features.
• Construct a region graph using object co-occurrence statistics to learn contextualized object labels via a graph neural network with gated updates.
• For each object pair, build a relationship graph with K relationship nodes representing co-occurrence probabilities and propagate messages to predict the relationship.
• Use a second graph neural network to model the interplay between object pair and potential relationships and output the final relation label.
• Embed prior statistics m_cc', m_cc'k into the graphs to explicitly regularize the semantic space, and evaluate using mR@K and R@K as complementary metrics.

Experimental results

Research questions

• RQ1 Can explicit, statistics-based knowledge graphs regularize the semantic space to improve relationship prediction, particularly for less frequent relationships?
• RQ2 Does integrating knowledge-embedded routing with graph propagation outperform prior scene graph generation methods on Visual Genome across SGGen, SGCls, and PredCls?
• RQ3 What is the impact of the knowledge graph components on overall performance and robustness to long-tail distributions?
• RQ4 How do mR@K and R@K metrics reflect improvements for both frequent and rare relationships?

Key findings

• The proposed KERN framework yields superior mR@K scores compared with prior methods, with mean mR@50/100 improving from 9.0% (SMN) to 11.7% (KERN) and mean mR@50/100 improving further in ablations.
• For R@K metrics, KERN achieves state-of-the-art mean recalls, e.g., mean R@50/100 of 44.1%/55.4% under constrained settings, surpassing SMN by small margins.
• Ablation studies show replacing the statistical correlations with uniform distributions degrades performance (mean mR from 11.7% to 7.9%; mean R from 44.1% to 40.6%), confirming the importance of the explicit knowledge integration.
• KERN demonstrates larger gains for rare/less frequent relationships, addressing the long-tail distribution issue inherent in Visual Genome.