[Paper Review] Convolutional Xformers for Vision

Pranav Jeevan, Amit sethi|arXiv (Cornell University)|Jan 25, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Convolutional Xformers for Vision (CXV), a hybrid architecture combining linear attention mechanisms with convolutional sub-layers to reduce GPU memory usage and data requirements while maintaining state-of-the-art accuracy in image classification. By replacing quadratic self-attention with linear alternatives like Performer and Linear Transformer, and embedding inductive bias via convolutions, CXV eliminates the need for class tokens and positional embeddings, achieving superior performance under low-resource conditions compared to ViTs, ResNets, and other vision transformers.

ABSTRACT

Vision transformers (ViTs) have found only limited practical use in processing images, in spite of their state-of-the-art accuracy on certain benchmarks. The reason for their limited use include their need for larger training datasets and more computational resources compared to convolutional neural networks (CNNs), owing to the quadratic complexity of their self-attention mechanism. We propose a linear attention-convolution hybrid architecture -- Convolutional X-formers for Vision (CXV) -- to overcome these limitations. We replace the quadratic attention with linear attention mechanisms, such as Performer, Nyströmformer, and Linear Transformer, to reduce its GPU usage. Inductive prior for image data is provided by convolutional sub-layers, thereby eliminating the need for class token and positional embeddings used by the ViTs. We also propose a new training method where we use two different optimizers during different phases of training and show that it improves the top-1 image classification accuracy across different architectures. CXV outperforms other architectures, token mixers (e.g. ConvMixer, FNet and MLP Mixer), transformer models (e.g. ViT, CCT, CvT and hybrid Xformers), and ResNets for image classification in scenarios with limited data and GPU resources (cores, RAM, power).

Motivation & Objective

• To address the high computational and data demands of vision transformers (ViTs), which limit their practical use despite strong accuracy.
• To reduce GPU memory consumption and training data requirements in image classification by replacing quadratic self-attention with linear attention mechanisms.
• To integrate inductive biases from convolutions to eliminate reliance on class tokens and positional embeddings.
• To develop a training strategy that improves convergence and accuracy using dual optimizers across training phases.
• To create a model that generalizes well under low-resource constraints—limited data and GPU memory—while matching or exceeding state-of-the-art performance.

Proposed method

• Replace the quadratic self-attention mechanism in vision transformers with linear attention mechanisms such as Performer, Nyströmformer, and Linear Transformer to reduce computational complexity.
• Integrate convolutional sub-layers within each block to provide inductive bias for image data, enabling spatial feature learning without sequence unrolling.
• Eliminate class tokens and positional embeddings by preserving 2D spatial structure throughout the network via convolutional operations.
• Use a single layer normalization per block instead of pre-normalization to improve training stability and performance.
• Propose Dual Optimizer Training (DualOpT), using two different optimizers in distinct training phases to enhance convergence and accuracy.
• Maintain input resolution and spatial structure across all layers, avoiding hierarchical downsampling or patchification.

Experimental results

Research questions

• RQ1 Can a hybrid architecture combining linear attention and convolutions achieve state-of-the-art image classification accuracy with significantly reduced GPU memory usage?
• RQ2 Does replacing class tokens and positional embeddings with convolutional inductive bias improve model performance under low-data and low-GPU conditions?
• RQ3 Can a dual-optimizer training strategy improve convergence speed and final accuracy across diverse architectures like ViTs, ResNets, and token mixers?
• RQ4 How does the performance of CXV compare to existing models like ViT, ResNet, FNet, MLP-Mixer, and ConvMixer in data- and resource-constrained settings?
• RQ5 To what extent do architectural choices such as normalization placement and attention mechanism selection affect model generalization and efficiency?

Key findings

• CXV outperforms ResNet-18, ResNet-34, ViT, CCT, CvT, and other vision transformers in image classification under low-data and low-GPU conditions.
• CXV achieves up to 3 percentage points higher top-1 accuracy when using RandAugment followed by post-training on un-augmented data.
• The hybrid Vision Linear Transformer (ViLT) variant consumes only half the GPU memory of other hybrid ViX models like ViN and ViP.
• Using DualOpT training improved convergence and accuracy across multiple architectures, including ViTs, ResNets, and token mixers.
• ConvMixer and WaveMix perform at par with transformers but with significantly lower GPU usage, indicating that inductive priors are critical for efficiency.
• CXV uses orders of magnitude fewer parameters and less GPU memory than ResNets while achieving better accuracy in low-resource regimes.