[Paper Review] Recognizing Vector Graphics without Rasterization

Xinyang Jiang, Lu Liu+4 more|arXiv (Cornell University)|Nov 5, 2021

Advanced Image and Video Retrieval Techniques11 citations

TL;DR

This paper proposes YOLaT, a CNN-free, end-to-end object detection framework for vector graphics that operates directly on textual vector descriptions without rasterization. By constructing a multi-graph to encode structural and spatial relationships and using a dual-stream graph neural network, YOLaT achieves higher average precision and significantly better efficiency than raster-based baselines, outperforming even ImageNet-pretrained two-stage models with 25× fewer parameters and 100× fewer FLOPs.

ABSTRACT

In this paper, we consider a different data format for images: vector graphics. In contrast to raster graphics which are widely used in image recognition, vector graphics can be scaled up or down into any resolution without aliasing or information loss, due to the analytic representation of the primitives in the document. Furthermore, vector graphics are able to give extra structural information on how low-level elements group together to form high level shapes or structures. These merits of graphic vectors have not been fully leveraged in existing methods. To explore this data format, we target on the fundamental recognition tasks: object localization and classification. We propose an efficient CNN-free pipeline that does not render the graphic into pixels (i.e. rasterization), and takes textual document of the vector graphics as input, called YOLaT (You Only Look at Text). YOLaT builds multi-graphs to model the structural and spatial information in vector graphics, and a dual-stream graph neural network is proposed to detect objects from the graph. Our experiments show that by directly operating on vector graphics, YOLaT out-performs raster-graphic based object detection baselines in terms of both average precision and efficiency.

Motivation & Objective

• To address the lack of research on vector graphics in computer vision, particularly for object detection.
• To overcome the limitations of rasterization, such as resolution loss, high memory cost, and information degradation.
• To develop an efficient, end-to-end detection pipeline that operates directly on vector graphics' textual representations.
• To preserve structural and spatial information in vector graphics that is lost during rasterization.
• To demonstrate that direct vector processing enables more precise and efficient object localization and classification.

Proposed method

• Transforms all vector primitives (lines, curves, circles) into a unified textual format for consistent processing.
• Constructs an undirected multi-graph to model spatial and structural relationships between primitives using node features and edge types.
• Introduces a dual-stream graph neural network: one stream processes primitive-level features, and the other encodes relative positional differences between connected nodes.
• Uses a learnable aggregation function that combines features from both streams, with edge-specific message passing for improved representation learning.
• Generates object proposals directly from the graph structure without regression, enabling precise bounding box prediction.
• Employs a dual-stream GNN with early aggregation and edge attribute modeling to enhance feature learning while minimizing over-smoothing and computational cost.

Experimental results

Research questions

• RQ1 Can object detection in vector graphics be performed effectively without rasterization, preserving high-resolution and structural information?
• RQ2 How does direct processing of textual vector commands compare to rasterization-based methods in terms of detection accuracy and efficiency?
• RQ3 What is the impact of modeling both primitive-level features and relative spatial relationships in a dual-stream GNN architecture for vector graphics recognition?
• RQ4 Can a GNN-based model outperform standard CNN-based detectors on vector graphics when trained from scratch, without ImageNet pretraining?
• RQ5 How does preserving fine-grained geometric details (e.g., transistor arrow directions) affect detection performance in complex diagrams?

Key findings

• YOLaT achieves 98.67% mAP on the diagram dataset and 95.82% on the floorplan dataset, outperforming all raster-based baselines.
• On the same datasets, YOLaT achieves 25× fewer parameters and 100× fewer FLOPs than a powerful ImageNet-pretrained two-stage detector.
• The ablation study confirms that the dual-stream GNN with position-wise edge features and feature difference modeling improves performance by 2.8% mAP over ablated variants.
• Visualizations show that YOLaT produces tighter, more accurate bounding boxes that align exactly with object borders, unlike raster-based models that produce imprecise, overspreading boxes.
• YOLaT successfully distinguishes fine geometric details such as transistor arrow directions, which are lost in low-resolution rasterized images.
• The method demonstrates superior generalization on complex vector graphics, detecting more objects and reducing false negatives compared to YOLOv3, especially under high IoU thresholds.