[Paper Review] Answering Questions about Data Visualizations using Efficient Bimodal Fusion

Kushal Kafle, Robik Shrestha+3 more|arXiv (Cornell University)|Aug 5, 2019

Multimodal Machine Learning Applications44 references4 citations

TL;DR

This paper proposes PReFIL, a novel bimodal fusion model for chart question answering (CQA) that combines image and language features through parallel recurrent fusion to achieve state-of-the-art performance. PReFIL surpasses prior methods and even human baselines on both DVQA and FigureQA datasets, demonstrating strong generalization and enabling table reconstruction via iterative question answering.

ABSTRACT

Chart question answering (CQA) is a newly proposed visual question answering (VQA) task where an algorithm must answer questions about data visualizations, e.g. bar charts, pie charts, and line graphs. CQA requires capabilities that natural-image VQA algorithms lack: fine-grained measurements, optical character recognition, and handling out-of-vocabulary words in both questions and answers. Without modifications, state-of-the-art VQA algorithms perform poorly on this task. Here, we propose a novel CQA algorithm called parallel recurrent fusion of image and language (PReFIL). PReFIL first learns bimodal embeddings by fusing question and image features and then intelligently aggregates these learned embeddings to answer the given question. Despite its simplicity, PReFIL greatly surpasses state-of-the art systems and human baselines on both the FigureQA and DVQA datasets. Additionally, we demonstrate that PReFIL can be used to reconstruct tables by asking a series of questions about a chart.

Motivation & Objective

• To address the limitations of existing VQA models in understanding data visualizations, which require fine-grained measurement, OCR, and handling of out-of-vocabulary terms.
• To develop a robust CQA system that generalizes across diverse chart types and question formats.
• To surpass state-of-the-art performance on benchmark CQA datasets like DVQA and FigureQA.
• To explore the feasibility of using CQA models for downstream tasks such as automatic table reconstruction from charts.
• To identify gaps in current datasets and propose a roadmap for more realistic, challenging CQA benchmarks.

Proposed method

• PReFIL employs parallel recurrent fusion of image and language features, learning joint bimodal embeddings at both low- and high-level representations.
• It uses a dual-stream architecture to encode image features (via CNN) and question features (via RNN), followed by early fusion through concatenation and gated recurrent units.
• The model performs multi-step reasoning without explicit attention or relational modules by leveraging hierarchical aggregation of fused embeddings.
• It leverages pre-trained visual and textual encoders to extract rich, discriminative features before fusion.
• For table reconstruction, PReFIL applies iterative QA using predefined question templates to extract bar counts, labels, and values from charts.
• The system is trained end-to-end on CQA datasets and evaluated using standard metrics like accuracy and F1 score.

Experimental results

Research questions

• RQ1 Can a simple yet effective bimodal fusion mechanism outperform complex attention-based models in chart question answering?
• RQ2 To what extent can a CQA model surpass human performance on structured chart understanding tasks?
• RQ3 Can iterative question answering using a CQA model reconstruct underlying data tables from visualizations with high fidelity?
• RQ4 How do OCR quality and out-of-vocabulary terms affect CQA performance, and can the model be robust to such challenges?
• RQ5 What are the key limitations of current CQA datasets, and how can future benchmarks better reflect real-world chart understanding?

Key findings

• PReFIL achieves 91.14% accuracy on the DVQA dataset and 90.88% on FigureQA, significantly outperforming prior state-of-the-art methods.
• On FigureQA, PReFIL exceeds human performance, achieving 90.88% accuracy compared to a human baseline of 88.5%.
• On DVQA, PReFIL with oracle OCR surpasses human performance across all question types, achieving 91.10% accuracy on novel test charts.
• Table reconstruction using Algorithm 1 achieves 90.79% overall accuracy on familiar charts and 91.10% on novel charts, with near-perfect shape prediction (99.97%).
• The model demonstrates robustness to out-of-vocabulary terms and complex reasoning, particularly when combined with high-quality OCR.
• The study reveals that OCR quality is a critical bottleneck, as performance drops significantly with lower-quality OCR models.