[Paper Review] Escaping the Big Data Paradigm with Compact Transformers

Ali Hassani, Steven Walton+4 more|arXiv (Cornell University)|Apr 12, 2021

Advanced Neural Network Applications48 references295 citations

TL;DR

The paper introduces compact vision transformers (ViT-Lite, CVT, and CCT) that can be trained from scratch on small datasets, achieving competitive or state-of-the-art accuracy with far fewer parameters and compute. It demonstrates data-efficient transformer models for CIFAR-10/100, Flowers-102, and ImageNet without large-scale pre-training.

ABSTRACT

With the rise of Transformers as the standard for language processing, and their advancements in computer vision, there has been a corresponding growth in parameter size and amounts of training data. Many have come to believe that because of this, transformers are not suitable for small sets of data. This trend leads to concerns such as: limited availability of data in certain scientific domains and the exclusion of those with limited resource from research in the field. In this paper, we aim to present an approach for small-scale learning by introducing Compact Transformers. We show for the first time that with the right size, convolutional tokenization, transformers can avoid overfitting and outperform state-of-the-art CNNs on small datasets. Our models are flexible in terms of model size, and can have as little as 0.28M parameters while achieving competitive results. Our best model can reach 98% accuracy when training from scratch on CIFAR-10 with only 3.7M parameters, which is a significant improvement in data-efficiency over previous Transformer based models being over 10x smaller than other transformers and is 15% the size of ResNet50 while achieving similar performance. CCT also outperforms many modern CNN based approaches, and even some recent NAS-based approaches. Additionally, we obtain a new SOTA result on Flowers-102 with 99.76% top-1 accuracy, and improve upon the existing baseline on ImageNet (82.71% accuracy with 29% as many parameters as ViT), as well as NLP tasks. Our simple and compact design for transformers makes them more feasible to study for those with limited computing resources and/or dealing with small datasets, while extending existing research efforts in data efficient transformers. Our code and pre-trained models are publicly available at https://github.com/SHI-Labs/Compact-Transformers.

Motivation & Objective

• Motivate and enable transformer models to train from scratch on small datasets where data is scarce.
• Develop compact Transformer variants that combine convolutional tokenization with attention for data efficiency and locality.
• Propose SeqPool to replace the class token and improve pooling of output token sequences.
• Show that CCT with convolutional tokenizers provides strong accuracy while maintaining low parameter counts and compute.
• Demonstrate state-of-the-art or competitive results on CIFAR-10/100, Flowers-102, and ImageNet relative to model size and data regime.

Proposed method

• Propose ViT-Lite, CVT, and CCT as compact vision transformer variants suitable for small-data regimes.
• Replace standard patch-based tokenization with a convolutional tokenizer in CCT to embed local structure.
• Introduce SeqPool, an attention-based sequence pooling mechanism to map transformer outputs to a single class representation.
• Evaluate models trained from scratch on CIFAR-10/100, CIFAR, MNIST, Fashion-MNIST, Flowers-102, and ImageNet-1k using AdamW with cosine annealing.
• Compare against CNNs and ViT/DeiT baselines, including distillation scenarios, and report parameter counts and MACs.

Experimental results

Research questions

• RQ1 Can vision transformers be effectively trained from scratch on small datasets without large pre-training?
• RQ2 Do compact transformer architectures with convolutional tokenization and sequence pooling offer data-efficient improvements over ViT and CNNs on small datasets?
• RQ3 What are the impacts of using a convolutional tokenizer and SeqPool on accuracy and efficiency across various image datasets?
• RQ4 How does CCT perform on medium-scale datasets like ImageNet compared to traditional CNNs and ViT variants?
• RQ5 Is it feasible to deploy transformer models on limited compute resources while maintaining competitive performance?

Key findings

• CCT achieves top results on CIFAR-10 with 98% accuracy when trained from scratch on a model with about 3.7M parameters (5000 epochs yields 98.00% on CIFAR-10 in Table 2).
• CCT outperforms ViT and many CNN-based approaches on CIFAR-10/100 and Flowers-102 while using far fewer parameters and MACs (e.g., CVT and CCT variants show strong results with 0.28–3.85M parameters).
• On ImageNet-1k, CCT-14/7×2 reaches 80.67% Top-1 without distillation and with 22.36M parameters, and Distilled CCT variants reach 81.34% Top-1.
• Flowers-102 results show CCT-14/7×2 achieving 99.76% Top-1 under ImageNet-scale pretraining, with substantially fewer parameters (~22.17M) and MACs (18.63G).
• CCT demonstrates superior data efficiency by reducing model size to roughly 15% of ResNet50 while attaining similar or better performance on CIFAR-10/100.