[Paper Review] A Question-Answering framework for plots using Deep learning.

Revanth Reddy, Rahul Ramesh+2 more|arXiv (Cornell University)|Jun 12, 2018

Topic Modeling14 references3 citations

TL;DR

This paper proposes a novel deep learning framework for question-answering on bar graphs and pie charts by jointly identifying plot elements, quantifying values, and determining value ordering. It outperforms Relation Networks and CNN-LSTM baselines on the FigureQA dataset with faster training (~2 days on 1 GPU), demonstrating improved reasoning and efficiency in visual-visual reasoning tasks.

ABSTRACT

Deep Learning has managed to push boundaries in a wide variety of tasks. One area of interest is to tackle problems in reasoning and understanding, in an aim to emulate human intelligence. In this work, we describe a deep learning model that addresses the reasoning task of question-answering on bar graphs and pie charts. We introduce a novel architecture that learns to identify various plot elements, quantify the represented values and determine a relative ordering of these statistical values. We test our model on the recently released FigureQA dataset, which provides images and accompanying questions, for bar graphs and pie charts, augmented with rich annotations. Our approach outperforms the state-of-the-art Relation Networks baseline and traditional CNN-LSTM models when evaluated on this dataset. Our model also has a considerably faster training time of approximately 2 days on 1 GPU compared to the Relation Networks baseline

Motivation & Objective

• To develop a deep learning model that reasons over visual plots such as bar graphs and pie charts.
• To improve performance on visual question-answering tasks involving quantitative reasoning on graphical data.
• To address limitations in existing models like Relation Networks and CNN-LSTM in understanding plot structure and value relationships.
• To reduce training time while maintaining or improving accuracy on visual QA benchmarks.

Proposed method

• The model employs a novel architecture designed to detect and localize plot elements such as bars, slices, labels, and axes.
• It quantifies represented values by estimating numerical magnitudes from visual features using end-to-end learning.
• The framework learns relative ordering of statistical values through structured reasoning over detected elements.
• It integrates visual feature extraction with reasoning modules to jointly optimize element detection and value inference.
• The architecture is trained end-to-end on the FigureQA dataset using supervised learning with rich annotations.
• Training is conducted on a single GPU, achieving a training time of approximately 2 days.

Experimental results

Research questions

• RQ1 Can a deep learning model effectively recognize and reason about plot elements in bar graphs and pie charts?
• RQ2 How does the proposed model compare in performance to Relation Networks and CNN-LSTM baselines on visual QA tasks?
• RQ3 To what extent does the model improve training efficiency while maintaining or improving accuracy?
• RQ4 Can the model accurately infer quantitative values and their relative ordering from visual plots?

Key findings

• The proposed model achieves superior performance compared to the Relation Networks baseline on the FigureQA dataset.
• It outperforms traditional CNN-LSTM models in reasoning about visual plots and answering complex questions.
• The model reduces training time to approximately 2 days when trained on a single GPU, significantly faster than the Relation Networks baseline.
• The framework demonstrates improved generalization in identifying plot components and inferring numerical values from visual representations.