[Paper Review] Scene Graph Parsing as Dependency Parsing

Yu-Siang Wang, Chenxi Liu+2 more|arXiv (Cornell University)|Mar 25, 2018

Multimodal Machine Learning Applications43 references7 citations

TL;DR

This paper proposes a novel end-to-end neural dependency parser for scene graph parsing by reinterpreting scene graphs as edge-centric dependency structures. By redesigning the label and action space to focus on objects, attributes, and relations, the model outperforms prior two-stage approaches, achieving a 49.67% F-score on scene graph parsing—5% higher than previous state-of-the-art—while demonstrating improved performance in image retrieval tasks.

ABSTRACT

In this paper, we study the problem of parsing structured knowledge graphs from textual descriptions. In particular, we consider the scene graph representation that considers objects together with their attributes and relations: this representation has been proved useful across a variety of vision and language applications. We begin by introducing an alternative but equivalent edge-centric view of scene graphs that connect to dependency parses. Together with a careful redesign of label and action space, we combine the two-stage pipeline used in prior work (generic dependency parsing followed by simple post-processing) into one, enabling end-to-end training. The scene graphs generated by our learned neural dependency parser achieve an F-score similarity of 49.67% to ground truth graphs on our evaluation set, surpassing best previous approaches by 5%. We further demonstrate the effectiveness of our learned parser on image retrieval applications.

Motivation & Objective

• To address the limitations of two-stage scene graph parsing pipelines that rely on generic dependency parsing followed by heuristic or simple classifier post-processing.
• To improve interpretability and performance by redefining scene graphs in an edge-centric view that aligns naturally with dependency parsing structures.
• To eliminate reliance on pre-trained linguistic dependency parsers and instead train a customized neural dependency parser directly on visual-linguistic scene graph data.
• To evaluate the parser’s effectiveness not only on F-score but also on downstream vision tasks like image retrieval.
• To explore the potential of using image-anchored scene graphs as a more robust and semantically meaningful representation than dense vector embeddings.

Proposed method

• Reformulate scene graphs as edge-centric dependency structures, where edges represent relationships (objects, attributes, relations) rather than nodes.
• Redesign the label space to include only three types: object, attribute, and relation, eliminating redundant linguistic labels from standard dependency parsing.
• Modify the action space to allow for non-head-attached nodes, enabling the parser to omit function words and focus on semantic content.
• Train a single, end-to-end neural dependency parser using the BIST framework (Kiperwasser and Goldberg, 2016), fine-tuned on the Visual Genome dataset with aligned word-to-node mappings.
• Use a supervised learning setup with gold-standard scene graph annotations to train the parser directly on scene graph structure, bypassing intermediate dependency trees.
• Apply the parser to image retrieval by treating parsed scene graphs as query representations and measuring retrieval performance via recall@5, recall@10, and median rank.

Experimental results

Research questions

• RQ1 Can scene graph parsing be unified with dependency parsing by reinterpreting scene graphs in an edge-centric manner?
• RQ2 Does a customized dependency parser with a reduced label and action space outperform two-stage pipelines combining generic dependency parsing and post-processing?
• RQ3 Can end-to-end training on visual-linguistic data improve scene graph parsing accuracy compared to transferring knowledge from linguistic treebanks?
• RQ4 To what extent can the learned parser generalize to downstream vision tasks like image retrieval?
• RQ5 How does the quality of the sentence-to-graph alignment affect the final parsing performance and generalization?

Key findings

• The proposed end-to-end neural dependency parser achieves an F-score of 49.67% on the scene graph parsing task, surpassing the best prior approach by 5 percentage points.
• The parser outperforms the Stanford Scene Graph Parser in image retrieval, achieving higher recall@5, recall@10, and median rank, indicating better generalization to vision applications.
• The oracle F-score of 69.85% on aligned training graphs suggests that further improvements are possible through better alignment or model architecture.
• Ablation studies confirm that the parser’s performance is robust to design choices such as the direction of CONT arcs, validating the stability of the architecture.
• The model’s performance gain is attributed to the joint optimization of parsing and scene graph structure, rather than relying on separate post-processing steps.
• The results indicate that grounding scene graphs to visual regions without considering attributes and relations (as in Johnson et al., 2015) is suboptimal, and parsing quality significantly impacts retrieval performance.