[Paper Review] Spatially-sparse convolutional neural networks

Benjamin Graham|arXiv (Cornell University)|Sep 22, 2014

Advanced Neural Network Applications12 references197 citations

TL;DR

This paper proposes spatially-sparse convolutional neural networks (CNNs) that exploit sparsity in inputs—such as one-pixel-wide pen strokes in online handwriting or padded images—to dramatically accelerate training and inference in deep CNNs. By leveraging sparse computation, the method enables efficient training of deeper architectures, achieving state-of-the-art results: 3.82% test error on CASIA-OLHWDB1.1 and 6.28% on CIFAR-10, outperforming prior methods with similar or reduced computational cost.

ABSTRACT

Convolutional neural networks (CNNs) perform well on problems such as handwriting recognition and image classification. However, the performance of the networks is often limited by budget and time constraints, particularly when trying to train deep networks. Motivated by the problem of online handwriting recognition, we developed a CNN for processing spatially-sparse inputs; a character drawn with a one-pixel wide pen on a high resolution grid looks like a sparse matrix. Taking advantage of the sparsity allowed us more efficiently to train and test large, deep CNNs. On the CASIA-OLHWDB1.1 dataset containing 3755 character classes we get a test error of 3.82%. Although pictures are not sparse, they can be thought of as sparse by adding padding. Applying a deep convolutional network using sparsity has resulted in a substantial reduction in test error on the CIFAR small picture datasets: 6.28% on CIFAR-10 and 24.30% for CIFAR-100.

Motivation & Objective

• To address the computational inefficiency of training deep CNNs on high-resolution or sparse inputs, such as online handwriting data.
• To enable the use of deeper, more accurate CNN architectures by exploiting sparsity to reduce training and inference time.
• To improve performance on image recognition tasks by combining sparsity with data augmentation and deeper network designs.
• To explore the feasibility of applying sparse CNNs to non-sparse images by adding padding, preserving sparsity benefits.

Proposed method

• The method uses a novel family of deep CNNs, DeepCNet(ℓ,k), with alternating 3×3 and 2×2 max-pooling layers, designed to preserve spatial sparsity through early layers.
• Sparsity is exploited via sparse tensor operations that skip zero-valued computations, significantly reducing FLOPs and memory usage.
• The input is represented as a sparse binary grid (e.g., one-pixel pen strokes on high-res canvas), enabling efficient processing even at large input sizes.
• The framework supports data augmentation via affine transformations and padding, with sparsity maintained throughout the network.
• A modified network, DeepCNiN, integrates Network-in-Network layers to improve feature learning while retaining sparsity.
• The method generalizes to non-sparse images by treating them as sparse through padding, enabling efficient full-mode convolutions.

Experimental results

Research questions

• RQ1 Can sparsity in input data be exploited to train deeper and more accurate CNNs efficiently?
• RQ2 How does sparsity affect the performance and training speed of deep CNNs on online handwriting recognition tasks?
• RQ3 Can sparse CNNs achieve state-of-the-art results on standard image benchmarks like CIFAR-10 and CIFAR-100?
• RQ4 Does sparsity enable more flexible data augmentation strategies, such as translation and affine transformations, without increasing computational cost?
• RQ5 Can sparse CNNs be extended to higher-dimensional data, such as 3D objects or space-time trajectories?

Key findings

• On the CASIA-OLHWDB1.1 dataset with 3755 character classes, the method achieved a test error of 3.82% using DeepCNet(6,100), outperforming prior work.
• In the ICDAR2013 Chinese handwriting competition, the method achieved a test error of 2.61%, winning first place and surpassing the second-best entry (3.13%).
• On MNIST, a DeepCNet(5,10) achieved 0.58% test error, and a deeper DeepCNet(5,60) with dropout achieved 0.31%, demonstrating high accuracy with sparse computation.
• On CIFAR-10, the method achieved 6.28% test error with DeepCNiN(5,300), improving upon baseline results and reducing error by 2.53% compared to non-sparse baselines.
• On CIFAR-100, the method achieved 24.30% test error, a significant improvement over the 29.81% error of the base DeepCNet and the 35.68% of non-augmented baselines.
• The method enables real-time inference at 3000 characters per second on a single GPU, demonstrating practical efficiency for low-power devices.