[Paper Review] Multi-Graph Transformer for Free-Hand Sketch Recognition

Peng Xu, Chaitanya K. Joshi+1 more|arXiv (Cornell University)|Dec 24, 2019

Advanced Image and Video Retrieval Techniques64 references34 citations

TL;DR

The paper represents sketches as sparsely connected graphs and introduces a Multi-Graph Transformer that uses multiple graph structures to capture local geometry and global temporal relations, achieving strong sketch recognition performance.

ABSTRACT

Learning meaningful representations of free-hand sketches remains a challenging task given the signal sparsity and the high-level abstraction of sketches. Existing techniques have focused on exploiting either the static nature of sketches with Convolutional Neural Networks (CNNs) or the temporal sequential property with Recurrent Neural Networks (RNNs). In this work, we propose a new representation of sketches as multiple sparsely connected graphs. We design a novel Graph Neural Network (GNN), the Multi-Graph Transformer (MGT), for learning representations of sketches from multiple graphs which simultaneously capture global and local geometric stroke structures, as well as temporal information. We report extensive numerical experiments on a sketch recognition task to demonstrate the performance of the proposed approach. Particularly, MGT applied on 414k sketches from Google QuickDraw: (i) achieves small recognition gap to the CNN-based performance upper bound (72.80% vs. 74.22%), and (ii) outperforms all RNN-based models by a significant margin. To the best of our knowledge, this is the first work proposing to represent sketches as graphs and apply GNNs for sketch recognition. Code and trained models are available at https://github.com/PengBoXiangShang/multigraph_transformer.

Motivation & Objective

• Motivate modeling free-hand sketches as graphs to better capture sparse, abstract stroke structures.
• Develop a Transformer-based architecture that can operate on multiple sparse graphs representing sketches.
• Demonstrate that the multi-graph transformer outperforms RNN-based models and competes with CNN-based methods on QuickDraw data.
• Showcase the approach’s robustness and potential applicability beyond sketching tasks.

Proposed method

• Represent each sketch as multiple sparsely connected graphs: intra-stroke (local) and extra-stroke/global (temporal) graphs.
• Introduce Multi-Graph Transformer (MGT) with a Multi-Graph Multi-Head Attention sub-layer and a position-wise feed-forward network.
• Compute node embeddings from multi-modal inputs including coordinates, pen-state flags, and temporal position encodings.
• Aggregate information across multiple graphs via a Multi-Graph Multi-Head Attention mechanism and sum node features for classification.

Experimental results

Research questions

• RQ1 Can sketches be effectively represented and processed as multiple sparse graphs for recognition?
• RQ2 Does a Transformer that operates on multiple graphs outperform RNN- and CNN-based sketch recognizers on large-scale data?
• RQ3 What is the contribution of local (intra-stroke) versus global (temporal) graph structures to recognition performance?
• RQ4 How does multi-modal input design (coordinates, pen state, temporal encoding) influence performance?
• RQ5 Is the approach scalable and beneficial for real-time sketch-based human-computer interaction?

Key findings

• MGT achieves 0.707 acc@1 on Base and 0.728 acc@1 on Large, outperforming several RNN baselines and approaching CNN upper bounds on Google QuickDraw 345-class classification.
• MGT Large narrows the gap to top CNNs (Inception V3 and MobileNet V2) with 0.728 acc@1 vs. 0.7422 and 0.7310 respectively.
• Using domain-specific sketch graphs (A1-hop, A2-hop, A-global) significantly improves performance over fully-connected or random graphs.
• A three-graph configuration that combines local intra-stroke and global temporal graphs yields the best results among ablations.
• Compared to GCN/GAT baselines, MGT with sketch-specific graphs and multi-graph attention provides clear accuracy gains.