[Paper Review] data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

Alexei Baevski, Wei-Ning Hsu+4 more|arXiv (Cornell University)|Feb 7, 2022

Domain Adaptation and Few-Shot LearningComputer Science240 citations

TL;DR

data2vec unifies self-supervised learning across speech, vision, and language by predicting contextualized latent representations from masked inputs using a teacher-student Transformer setup, achieving competitive or state-of-the-art results across benchmarks.

ABSTRACT

While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech, NLP or computer vision. The core idea is to predict latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which are local in nature, data2vec predicts contextualized latent representations that contain information from the entire input. Experiments on the major benchmarks of speech recognition, image classification, and natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.

Motivation & Objective

• Motivate a single self-supervised learning framework applicable to multiple modalities (speech, vision, language).
• Develop a masking-based prediction task that targets contextual latent representations rather than fixed local targets.
• Demonstrate that predicting averaged latent representations from multiple teacher layers improves cross-modal performance.
• Show state-of-the-art or competitive results on major benchmarks in vision (ImageNet), speech (ASR), and NLP (GLUE).
• Provide insights into the effects of target layering, context, and modality-specific design choices on performance.

Proposed method

• Use a standard Transformer architecture operated in teacher and student modes; teacher is an EMA of the student.
• Encode full-input representations as targets using the teacher; encode masked inputs with the student to predict those targets.
• Construct targets by averaging normalized representations from the top K teacher blocks for masked time-steps.
• Train with a Smooth L1 loss to regress contextualized targets from masked predictions.
• Employ modality-specific input encoders and masking strategies (vision patches, speech spans, word/subword masking) to suit each modality.
• Evaluate two model sizes (Base and Large) across ImageNet, Librispeech, and GLUE-style benchmarks.

Experimental results

Research questions

• RQ1 Can a single self-supervised learning objective be effective across vision, speech, and language without modality-specific targets?
• RQ2 Do contextualized latent targets and multi-layer target averaging improve downstream performance across modalities?
• RQ3 How do context size and target feature choices affect learning and transfer to downstream tasks?
• RQ4 Does the data2vec approach yield state-of-the-art or competitive results on standard benchmarks in each modality?

Key findings

• Data2vec achieves competitive or state-of-the-art results on major benchmarks across vision, speech, and language.
• Predicting contextualized latent representations and averaging multiple teacher layers often yields better downstream accuracy than predicting a single top-layer representation, across all modalities.
• Contextualized targets require full input context for best performance; restricting the teacher’s context degrades results.
• Using FFN outputs (rather than self-attention outputs) as targets tends to perform better for speech pre-training.
• The unified framework enables strong performance without relying on discrete target tokens (e.g., words, visual tokens) in NLP and still delivers robust results in vision and speech.