[Paper Review] Graph Edit Distance Reward: Learning to Edit Scene Graph

Lichang Chen, Guosheng Lin+2 more|arXiv (Cornell University)|Aug 15, 2020

Multimodal Machine Learning Applications32 references4 citations

TL;DR

This paper proposes a Graph Edit Distance (GED) reward mechanism to train a neural symbolic model for learning to edit scene graphs based on natural language instructions in text-editing image retrieval. By leveraging policy gradient optimization with GED-based dense rewards, the model outperforms existing methods on both the CSS and a newly proposed CRIR dataset, demonstrating superior generalization and accuracy in complex, multi-hop reasoning scenarios.

ABSTRACT

Scene Graph, as a vital tool to bridge the gap between language domain and image domain, has been widely adopted in the cross-modality task like VQA. In this paper, we propose a new method to edit the scene graph according to the user instructions, which has never been explored. To be specific, in order to learn editing scene graphs as the semantics given by texts, we propose a Graph Edit Distance Reward, which is based on the Policy Gradient and Graph Matching algorithm, to optimize neural symbolic model. In the context of text-editing image retrieval, we validate the effectiveness of our method in CSS and CRIR dataset. Besides, CRIR is a new synthetic dataset generated by us, which we will publish it soon for future use.

Motivation & Objective

• To address the gap in learning to edit scene graphs according to user-provided text instructions in cross-modal image retrieval.
• To improve upon coarse 0/1 rewards in neural symbolic models by introducing a more precise, dense reward signal based on Graph Edit Distance (GED).
• To develop a new synthetic dataset, CRIR, that supports complex, multi-hop reasoning and reduces reliance on exact spatial location constraints.
• To enable more generalizable and explainable image retrieval by transforming image editing into symbolic scene graph editing with neural program generation.
• To validate the effectiveness of GED-based reward in enhancing model performance on both simple and complex query types.

Proposed method

• Propose a Graph Edit Distance (GED) reward function that quantifies the difference between predicted and target scene graphs, incorporating node insertion/deletion and attribute/edge substitution costs.
• Integrate the GED reward into a policy gradient framework to fine-tune a neural program generator that produces symbolic edit operations (e.g., add, remove, modify) on scene graphs.
• Design a scene graph generation engine to parse images into symbolic representations with nodes (objects) and edges (relations), enabling structured reasoning.
• Implement a modifying engine that applies program outputs to update the scene graph, including adding pseudo-nodes with full edge connections and removing nodes with all incident edges.
• Use a seq2seq program generator to predict latent programs from input text, with length normalization via padding to handle variable-length instructions.
• Apply early stopping and batch training with a fixed batch size of 64 over 60,000 iterations to optimize the program generator using the GED reward.

Experimental results

Research questions

• RQ1 Can a dense, differentiable reward based on Graph Edit Distance improve the learning of scene graph editing compared to binary 0/1 rewards?
• RQ2 How effective is the proposed method in handling complex, multi-hop reasoning queries that do not rely on exact spatial locations?
• RQ3 Does the GED-based reward lead to better generalization and performance on unseen, complex image retrieval tasks?
• RQ4 Can the model generalize beyond datasets with rigid spatial constraints, such as CSS, to more realistic, relational scene graph scenarios?
• RQ5 To what extent does the proposed CRIR dataset enable evaluation of advanced reasoning and editing capabilities in text-editing image retrieval?

Key findings

• The proposed model with GED reward achieves state-of-the-art performance on both the CSS and CRIR datasets, surpassing TIRG and Concat models across all query types.
• On the CRIR dataset, the model maintains high performance even on complex three-hop queries, where TIRG and Concat models fail significantly.
• The GED reward leads to more efficient and accurate learning, as evidenced by superior performance in the last two columns of Table 2, indicating better alignment with target graphs.
• The model successfully retrieves correct images in qualitative cases involving relational edits (e.g., adding an object to the left of another), while TIRG fails by retrieving images based only on visual similarity.
• The CRIR dataset, generated using the CLEVR toolkit, effectively simulates real-world complexity by removing exact location constraints and enabling multi-hop reasoning.
• The model demonstrates strong generalization, particularly in relational editing tasks, where it outperforms baselines that rely on feature-level fusion or simpler rewards.