[Paper Review] Effective Vision Transformer Training: A Data-Centric Perspective

Benjia Zhou, Pichao Wang+3 more|arXiv (Cornell University)|Sep 29, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a data-centric training framework for Vision Transformers (ViTs) that dynamically measures and generates 'effective' training examples by adjusting sample difficulty across three training stages—formation, growth, and exploration. It introduces a dynamic MixUp strategy and a patch-level erasing method (PatchErasing) to enhance generalization, achieving consistent accuracy gains of ≥0.5% on DeiT and ≥0.4% on Swin Transformers across model sizes with minimal training overhead.

ABSTRACT

Vision Transformers (ViTs) have shown promising performance compared with Convolutional Neural Networks (CNNs), but the training of ViTs is much harder than CNNs. In this paper, we define several metrics, including Dynamic Data Proportion (DDP) and Knowledge Assimilation Rate (KAR), to investigate the training process, and divide it into three periods accordingly: formation, growth and exploration. In particular, at the last stage of training, we observe that only a tiny portion of training examples is used to optimize the model. Given the data-hungry nature of ViTs, we thus ask a simple but important question: is it possible to provide abundant ``effective'' training examples at EVERY stage of training? To address this issue, we need to address two critical questions, \ie, how to measure the ``effectiveness'' of individual training examples, and how to systematically generate enough number of ``effective'' examples when they are running out. To answer the first question, we find that the ``difficulty'' of training samples can be adopted as an indicator to measure the ``effectiveness'' of training samples. To cope with the second question, we propose to dynamically adjust the ``difficulty'' distribution of the training data in these evolution stages. To achieve these two purposes, we propose a novel data-centric ViT training framework to dynamically measure the ``difficulty'' of training samples and generate ``effective'' samples for models at different training stages. Furthermore, to further enlarge the number of ``effective'' samples and alleviate the overfitting problem in the late training stage of ViTs, we propose a patch-level erasing strategy dubbed PatchErasing. Extensive experiments demonstrate the effectiveness of the proposed data-centric ViT training framework and techniques.

Motivation & Objective

• To investigate why ViT training suffers from suboptimal performance despite large-scale data, particularly due to insufficient effective examples in later training stages.
• To identify and quantify the existence of three distinct training evolution stages: formation, growth, and exploration.
• To address the scarcity of effective training examples in later stages by measuring sample 'effectiveness' via difficulty and dynamically generating high-quality examples.
• To alleviate late-stage overfitting in ViTs through a novel patch-level erasing strategy (PatchErasing) that regularizes local patch tokens.
• To develop a unified, data-centric framework that flexibly adapts data difficulty based on model capacity and training phase.

Proposed method

• The authors define Dynamic Data Proportion (DDP) and Knowledge Assimilation Rate (KAR) to objectively identify three training stages: formation (easy samples), growth (moderate samples), and exploration (hard samples).
• Sample 'effectiveness' is measured by 'difficulty', with harder samples considered more effective for knowledge assimilation in later stages.
• A dynamic MixUp technique is proposed to adjust the difficulty distribution of training data according to the current training stage, enabling flexible generation of effective examples.
• PatchErasing is introduced as a patch-level data augmentation strategy that randomly erases image patches and regularizes the corresponding label vector to reduce overfitting to local tokens.
• The framework applies static difficulty scheduling for large models (e.g., DeiT-B, Swin-B) and dynamic scheduling for small models (e.g., DeiT-T, DeiT-S) based on their learning capacity and risk of local optima.
• The method is integrated into standard ViT training with no significant computational overhead, enabling consistent performance gains across DeiT and Swin Transformer architectures.

Experimental results

Research questions

• RQ1 Can the ViT training process be systematically divided into distinct evolutionary stages based on data utilization and model learning dynamics?
• RQ2 How can the 'effectiveness' of individual training examples be measured during ViT training, particularly in relation to model capacity and training phase?
• RQ3 Can dynamic adjustment of training sample difficulty improve the number of effective examples across all training stages, especially in the late exploration phase?
• RQ4 Does patch-level data augmentation (PatchErasing) reduce overfitting to local image patches and improve generalization in ViTs?
• RQ5 Can a unified data-centric training framework consistently improve ViT performance across different model sizes with minimal training cost?

Key findings

• The ViT training process naturally divides into three stages—formation, growth, and exploration—based on Dynamic Data Proportion (DDP) and Knowledge Assimilation Rate (KAR), with distinct data utilization patterns.
• In the final exploration stage, only a small fraction of training examples (hard and moderate) contribute to optimization, while easy samples are largely ignored, indicating a critical bottleneck in data utilization.
• The proposed dynamic difficulty adjustment via MixUp improves performance by +0.4% on DeiT-T, +0.5% on DeiT-S, and +0.5% on DeiT-B, with better results for small models due to flexible scheduling.
• PatchErasing achieves a +0.4% performance gain on large models (e.g., DeiT-B) when using a 50% mask rate, demonstrating its effectiveness in reducing patch-level overfitting.
• The framework consistently improves accuracy across both global-based (DeiT) and local-based (Swin) ViT architectures, with gains of ≥0.5% on DeiT and ≥0.4% on Swin Transformers.
• Visualizations show that PatchErasing encourages models to attend to more object regions, reducing reliance on a few salient patches and improving robustness.