[Paper Review] Turbo Training with Token Dropout

Tengda Han, Weidi Xie+1 more|arXiv (Cornell University)|Oct 10, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Turbo training, a computation-efficient paradigm for training Vision Transformers on video tasks by jointly optimizing a masked autoencoding loss and a downstream task loss using sparsely sampled visual tokens. The method achieves nearly 4× faster training and reduced memory usage while maintaining competitive performance on action classification, video-language learning, and long-video activity recognition, enabling previously infeasible end-to-end long-video training.

ABSTRACT

The objective of this paper is an efficient training method for video tasks. We make three contributions: (1) We propose Turbo training, a simple and versatile training paradigm for Transformers on multiple video tasks. (2) We illustrate the advantages of Turbo training on action classification, video-language representation learning, and long-video activity classification, showing that Turbo training can largely maintain competitive performance while achieving almost 4X speed-up and significantly less memory consumption. (3) Turbo training enables long-schedule video-language training and end-to-end long-video training, delivering competitive or superior performance than previous works, which were infeasible to train under limited resources.

Motivation & Objective

• Address the high computational and memory cost of training Vision Transformers on long video data, which hinders research progress and limits accessibility.
• Overcome the bottleneck of long training schedules and resource demands in video understanding, especially for researchers with limited hardware.
• Enable end-to-end training of Transformers on long videos by reducing the need for pre-extracted features or two-stage training pipelines.
• Leverage the redundancy in video data to maintain performance while drastically reducing the number of tokens processed during training.
• Demonstrate that joint pre-training via masked autoencoding and downstream task optimization can yield strong representations with minimal computational overhead.

Proposed method

• Apply sparse token dropout during training, where a high proportion of spatio-temporal tokens (up to 90%) are masked and reconstructed via a masked autoencoder.
• Train the model end-to-end with a multi-task objective: a masked autoencoding loss for self-supervised representation learning and a standard downstream loss (e.g., cross-entropy or contrastive loss).
• Use a fixed masking ratio per training step, with frame sampling restricted to the first and last 20% of the video to ensure coverage of key moments.
• Maintain the same number of visible patches across different input lengths (e.g., 16, 32, 64 frames) by adjusting the masking ratio to balance computational cost and temporal context.
• Train the Vision Transformer encoder with both masked autoencoding and downstream task supervision in a single optimization loop, enabling joint feature learning.
• Utilize the inherent redundancy in video data—where only a sparse subset of tokens is needed to reconstruct the full signal—enabling efficient training without performance degradation.

Experimental results

Research questions

• RQ1 Can a masked autoencoding objective combined with downstream task supervision maintain competitive performance on video understanding tasks while drastically reducing computational cost?
• RQ2 To what extent can token dropout (e.g., 87.5% masking) preserve representation quality in long video tasks such as action classification and video-language alignment?
• RQ3 Can Turbo training enable end-to-end training of Transformers on long videos, bypassing the need for two-stage pipelines with pre-extracted features?
• RQ4 How does the temporal extent of input (e.g., 16 vs. 64 frames) affect performance when masked autoencoding is used with varying masking ratios?
• RQ5 Does joint video-language pre-training via Turbo training yield better-quality representations than methods relying on pre-extracted features or separate pre-training stages?

Key findings

• Turbo training achieves nearly 4× faster training speed and significant memory reduction while maintaining competitive performance across multiple video tasks.
• On the Breakfast dataset, Turbo training with 32-frame input and 75% masking ratio (F32) achieved 91.3% accuracy, outperforming prior two-stage methods (89.9%) that used pre-extracted features.
• On the COIN dataset, Turbo training achieved 87.5% accuracy with 32-frame input, comparable to the state-of-the-art method (88.9%) that used a two-stage training pipeline.
• The method enabled end-to-end training on long videos, such as in the HTM-AA dataset, where it outperformed previous approaches in video-language alignment (MIL-NCE: 31.3, IP: 22.0).
• Increasing the number of frames from 16 to 32 improved performance (88.2% to 91.3% on Breakfast), but further increasing to 64 frames (with 87.5% masking) reduced performance, suggesting that high masking ratios may miss key actions.
• Joint video-language pre-training via Turbo training produced high-quality embeddings, with 49.4% alignment accuracy on the TAN benchmark, outperforming methods that rely on pre-extracted features.