[Paper Review] GRCNN: Graph Recognition Convolutional Neural Network for Synthesizing Programs from Flow Charts

Lin Cheng, Zijiang Yang|arXiv (Cornell University)|Nov 11, 2020

Software Engineering Research4 citations

TL;DR

This paper proposes GRCNN, a deep convolutional neural network that synthesizes programs from flow chart images by jointly recognizing nodes and edges in the graph structure. Trained end-to-end, GRCNN achieves 66.4% program synthesis accuracy on synthetic data and 63.6% on real-world textbook examples, with 94.1% edge recognition and 67.9% node recognition accuracy, processing each flow chart in ~60 milliseconds.

ABSTRACT

Program synthesis is the task to automatically generate programs based on user specification. In this paper, we present a framework that synthesizes programs from flow charts that serve as accurate and intuitive specifications. In order doing so, we propose a deep neural network called GRCNN that recognizes graph structure from its image. GRCNN is trained end-to-end, which can predict edge and node information of the flow chart simultaneously. Experiments show that the accuracy rate to synthesize a program is 66.4%, and the accuracy rates to recognize edge and nodes are 94.1% and 67.9%, respectively. On average, it takes about 60 milliseconds to synthesize a program.

Motivation & Objective

• To enable program synthesis from intuitive, visual flow charts instead of formal or example-based specifications.
• To address the challenge of accurately recognizing graph structures (nodes and edges) from flow chart images using deep learning.
• To develop a unified, end-to-end deep learning framework that jointly predicts node content and edge connectivity from a single image input.
• To evaluate the feasibility and performance of using flow charts as accurate and user-friendly specifications for program synthesis.

Proposed method

• GRCNN employs a shared backbone convolutional neural network to extract a common feature representation from a resized flow chart image.
• The network uses a detection head to localize nodes via bounding boxes and a recognition head to predict node text content using a sequence-based decoder.
• An edge prediction head generates an adjacency matrix by predicting connections between detected nodes using a classification head over all node pairs.
• The model is trained end-to-end using a joint loss function combining node detection, text recognition, and edge classification losses.
• A spatial transformer network (STN) is applied to improve localization robustness, though ablation shows it slightly reduces edge accuracy.
• The system compiles the predicted graph structure into source code using a predefined mapping from node types and edge connections.

Experimental results

Research questions

• RQ1 Can a deep neural network accurately recognize both node content and edge connections in flow chart images in an end-to-end manner?
• RQ2 How effective is GRCNN in synthesizing correct programs from flow chart specifications compared to traditional program synthesis methods?
• RQ3 To what extent does sharing the backbone feature representation between node and edge heads improve efficiency without sacrificing accuracy?
• RQ4 How does the model perform on real-world, textbook-based flow charts compared to synthetic datasets?
• RQ5 What is the impact of architectural enhancements like STN on the recognition accuracy of nodes and edges?

Key findings

• GRCNN achieves a program synthesis accuracy of 66.4% on a synthetic dataset and 63.6% on a real-world textbook-based dataset.
• Edge recognition accuracy reaches 94.1% on synthetic data and 72.7% on real-world data, indicating strong structural understanding.
• Node recognition accuracy is 90.6% on synthetic data and 81.8% on real-world data, with 67.9% of images having all nodes correctly predicted.
• The average inference time for program synthesis is approximately 60 milliseconds, demonstrating real-time feasibility.
• Ablation studies show that while STN enhancements improve node detection and recognition, they slightly reduce edge accuracy, lowering overall graph accuracy.
• End-to-end training with shared backbone features reduces inference time by 5.8% (3.5ms) compared to separate networks, confirming computational efficiency.