[Paper Review] ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning

Fanqing Meng, Wenqi Shao+5 more|arXiv (Cornell University)|Jan 4, 2024

Topic Modeling4 citations

TL;DR

ChartAssistant is a universal multimodal language model for chart understanding, leveraging chart-to-table pre-training and multitask instruction tuning on a diverse dataset (ChartSFT) to achieve state-of-the-art performance across multiple chart tasks. It outperforms SOTA models like UniChart and ChartLlama, especially in zero-shot settings on real-world charts, with significant gains in numerical reasoning and generalization.

ABSTRACT

Charts play a vital role in data visualization, understanding data patterns, and informed decision-making. However, their unique combination of graphical elements (e.g., bars, lines) and textual components (e.g., labels, legends) poses challenges for general-purpose multimodal models. While vision-language models trained on chart data excel in comprehension, they struggle with generalization. To address these challenges, we propose ChartAssistant, a chart-based vision-language model for universal chart comprehension and reasoning. ChartAssistant leverages ChartSFT, a comprehensive dataset covering diverse chart-related tasks with basic (e.g. bars and pies) and specialized (e.g. radars, and bubbles) chart types. It undergoes a two-stage training process, starting with pre-training on chart-to-table parsing to align chart and text, followed by multitask instruction-following fine-tuning. This approach enables ChartAssistant to achieve competitive performance across various chart tasks. Experimental results demonstrate significant performance gains over the state-of-the-art UniChart and Chartllama method, especially outperforming them on real-world chart data with zero-shot setting. The code and data are available at https://github.com/OpenGVLab/ChartAst.

Motivation & Objective

• To address the limitations of general-purpose vision-language models in understanding complex charts with graphical and textual elements.
• To improve zero-shot generalization and reasoning in chart comprehension by aligning visual and structured textual representations.
• To overcome the poor generalization of existing chart models that require task-specific fine-tuning.
• To develop a comprehensive, diverse, and high-quality instruction-tuning dataset (ChartSFT) covering various chart types and tasks.
• To enable a single unified model to perform multiple chart-related tasks with high performance across diverse real-world scenarios.

Proposed method

• Pre-train the model on a chart-to-table translation task to align visual chart representations with structured textual tables.
• Construct ChartSFT, a large-scale instruction-following dataset with diverse chart types (e.g., radar, box plots), tasks (QA, summarization, referring), and annotations (including chain-of-thought for numerical reasoning).
• Perform multitask instruction tuning on ChartSFT, integrating multiple chart-related tasks into a single unified training objective.
• Use two model variants: ChartAst-D (260M parameters, based on Donut) and ChartAst-S (13B parameters, based on SPHINX) for efficiency and robustness.
• Incorporate chain-of-thought (CoT) annotations in training to enhance mathematical reasoning and numerical question-answering accuracy.
• Leverage GPT-3.5 to generate high-quality, diverse, and precise question-answer pairs, especially from arXiv for open-ended reasoning tasks.

Experimental results

Research questions

• RQ1 Can a unified vision-language model achieve strong zero-shot generalization across diverse chart types and tasks without task-specific fine-tuning?
• RQ2 How does chart-to-table pre-training improve alignment between visual and textual representations in charts?
• RQ3 To what extent does multitask instruction tuning with diverse chart tasks enhance reasoning and comprehension capabilities?
• RQ4 How effective are chain-of-thought annotations in improving numerical question-answering performance?
• RQ5 Can a single model trained on a broad, diverse dataset outperform task-specific fine-tuned models on real-world chart data?

Key findings

• ChartAssistant achieves state-of-the-art performance on all evaluated chart tasks, outperforming UniChart and ChartLlama, especially in zero-shot settings on real-world data.
• The use of chain-of-thought (CoT) annotations increases numerical question-answering accuracy from 51.9% to 72.1%, with gains of 22% and 26.6% on dot-line and line charts, respectively.
• Excluding any multitask instruction tuning component leads to performance degradation, with numerical question answering being the most critical component for overall performance.
• ChartAst-S (13B parameter variant) demonstrates superior robustness and performance due to dynamic resolution processing and mixed visual encoders inherited from SPHINX.
• The model achieves 91.3% accuracy on the ChartQA (augmented) benchmark and 72.1% on numerical QA, significantly outperforming prior SOTA methods.
• The ablation study confirms that the inclusion of diverse chart types (e.g., radar, box plots) and comprehensive annotations enhances generalization and model capability.