[论文解读] Multi-Graph Transformer for Free-Hand Sketch Recognition

Peng Xu, Chaitanya K. Joshi+1 more|arXiv (Cornell University)|Dec 24, 2019

Advanced Image and Video Retrieval Techniques参考文献 64被引用 34

一句话总结

本文将草图表示为稀疏连接图并引入一个多图变换器，利用多种图结构来捕捉局部几何和全局时间关系，在草图识别中实现了强大的性能。

ABSTRACT

Learning meaningful representations of free-hand sketches remains a challenging task given the signal sparsity and the high-level abstraction of sketches. Existing techniques have focused on exploiting either the static nature of sketches with Convolutional Neural Networks (CNNs) or the temporal sequential property with Recurrent Neural Networks (RNNs). In this work, we propose a new representation of sketches as multiple sparsely connected graphs. We design a novel Graph Neural Network (GNN), the Multi-Graph Transformer (MGT), for learning representations of sketches from multiple graphs which simultaneously capture global and local geometric stroke structures, as well as temporal information. We report extensive numerical experiments on a sketch recognition task to demonstrate the performance of the proposed approach. Particularly, MGT applied on 414k sketches from Google QuickDraw: (i) achieves small recognition gap to the CNN-based performance upper bound (72.80% vs. 74.22%), and (ii) outperforms all RNN-based models by a significant margin. To the best of our knowledge, this is the first work proposing to represent sketches as graphs and apply GNNs for sketch recognition. Code and trained models are available at https://github.com/PengBoXiangShang/multigraph_transformer.

研究动机与目标

• Motivate modeling free-hand sketches as graphs to better capture sparse, abstract stroke structures.
• Develop a Transformer-based architecture that can operate on multiple sparse graphs representing sketches.
• Demonstrate that the multi-graph transformer outperforms RNN-based models and competes with CNN-based methods on QuickDraw data.
• Showcase the approach’s robustness and potential applicability beyond sketching tasks.

提出的方法

• Represent each sketch as multiple sparsely connected graphs: intra-stroke (local) and extra-stroke/global (temporal) graphs.
• Introduce Multi-Graph Transformer (MGT) with a Multi-Graph Multi-Head Attention sub-layer and a position-wise feed-forward network.
• Compute node embeddings from multi-modal inputs including coordinates, pen-state flags, and temporal position encodings.
• Aggregate information across multiple graphs via a Multi-Graph Multi-Head Attention mechanism and sum node features for classification.

实验结果

研究问题

• RQ1 Can sketches be effectively represented and processed as multiple sparse graphs for recognition?
• RQ2 Does a Transformer that operates on multiple graphs outperform RNN- and CNN-based sketch recognizers on large-scale data?
• RQ3 What is the contribution of local (intra-stroke) versus global (temporal) graph structures to recognition performance?
• RQ4 How does multi-modal input design (coordinates, pen state, temporal encoding) influence performance?
• RQ5 Is the approach scalable and beneficial for real-time sketch-based human-computer interaction?

主要发现

• MGT achieves 0.707 acc@1 on Base and 0.728 acc@1 on Large, outperforming several RNN baselines and approaching CNN upper bounds on Google QuickDraw 345-class classification.
• MGT Large narrows the gap to top CNNs (Inception V3 and MobileNet V2) with 0.728 acc@1 vs. 0.7422 and 0.7310 respectively.
• Using domain-specific sketch graphs (A1-hop, A2-hop, A-global) significantly improves performance over fully-connected or random graphs.
• A three-graph configuration that combines local intra-stroke and global temporal graphs yields the best results among ablations.
• Compared to GCN/GAT baselines, MGT with sketch-specific graphs and multi-graph attention provides clear accuracy gains.