[Paper Review] ChartDETR: A Multi-shape Detection Network for Visual Chart Recognition

Wenyuan Xue, Dapeng Chen+4 more|arXiv (Cornell University)|Aug 15, 2023

Handwritten Text Recognition TechniquesComputer Science3 citations

TL;DR

ChartDETR is a transformer-based multi-shape detection network that predicts all data element shapes in a chart image in a single forward pass using query groups and keypoint set prediction, eliminating post-processing. It achieves state-of-the-art F1 scores of 0.98 on Adobe Synthetic and 0.97 on ExcelChart400k by directly regressing keypoints for bars, lines, and pies without additional design or grouping heuristics.

ABSTRACT

Visual chart recognition systems are gaining increasing attention due to the growing demand for automatically identifying table headers and values from chart images. Current methods rely on keypoint detection to estimate data element shapes in charts but suffer from grouping errors in post-processing. To address this issue, we propose ChartDETR, a transformer-based multi-shape detector that localizes keypoints at the corners of regular shapes to reconstruct multiple data elements in a single chart image. Our method predicts all data element shapes at once by introducing query groups in set prediction, eliminating the need for further postprocessing. This property allows ChartDETR to serve as a unified framework capable of representing various chart types without altering the network architecture, effectively detecting data elements of diverse shapes. We evaluated ChartDETR on three datasets, achieving competitive results across all chart types without any additional enhancements. For example, ChartDETR achieved an F1 score of 0.98 on Adobe Synthetic, significantly outperforming the previous best model with a 0.71 F1 score. Additionally, we obtained a new state-of-the-art result of 0.97 on ExcelChart400k. The code will be made publicly available.

Motivation & Objective

• To address the challenge of detecting diverse chart shapes (bar, line, pie) in a unified framework without post-processing.
• To overcome grouping errors in keypoint-based methods by eliminating the need for hand-designed post-processing rules.
• To enable end-to-end detection of multiple data elements using set prediction with query groups.
• To achieve high-accuracy detection across various chart types without architectural modifications.
• To provide a simple yet effective solution that generalizes across different chart types and visual appearances.

Proposed method

• ChartDETR uses a transformer-based set prediction framework with query groups to predict multiple data element shapes simultaneously.
• Each shape is represented by N keypoints, and the model predicts M×N keypoints in total using M query groups, each responsible for detecting one shape.
• A duplicated keypoint matching strategy interpolates ground-truth keypoints to match the predicted N points, enabling point-wise supervision.
• The model employs a bipartite matching loss between predicted keypoint groups and ground-truth shapes to ensure correct correspondence.
• A hybrid one-to-many matching scheme with K repeated ground-truth instances improves training stability and performance.
• The method is architecture-agnostic and works with various backbones, including ResNet, Swin, and HRNet, enabling flexibility and scalability.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework detect multiple chart types (bar, line, pie) without architectural changes or post-processing?
• RQ2 How does query grouping improve multi-shape detection compared to single-query detectors in charts with complex spatial layouts?
• RQ3 To what extent does duplicated keypoint matching and bipartite matching improve keypoint localization accuracy and reduce false positives?
• RQ4 What is the optimal configuration of N (keypoints per shape) and M (maximum number of shapes) for robust detection across diverse chart datasets?
• RQ5 How does the one-to-many matching strategy with K repeated ground-truth instances affect model convergence and performance?

Key findings

• ChartDETR achieved an F1 score of 0.98 on the Adobe Synthetic dataset, significantly outperforming the previous best model (0.71 F1).
• On the ExcelChart400k benchmark, ChartDETR achieved a new state-of-the-art F1 score of 0.97, demonstrating strong generalization across real-world chart images.
• The ablation study showed that setting N=14 (default) and M=8 yielded the best performance on Adobe Synthetic, with further increases in M or N leading to performance degradation.
• The one-to-many matching scheme with K=3 provided the highest performance gain, improving F1 by 0.029 over one-to-one matching alone.
• HRNet backbones outperformed ResNet and Swin variants on bar detection, achieving 0.923 F1, indicating that high-resolution feature maps are crucial for detecting small, crowded bars.
• The method achieved consistent performance across all chart types without architectural modifications, proving its robustness and generalization capability.