[Paper Review] Masked Frequency Modeling for Self-Supervised Visual Pre-Training

Jiahao Xie, Wei Li+4 more|arXiv (Cornell University)|Jun 15, 2022

Image Processing Techniques and Applications28 citations

TL;DR

MFM masks frequency components in the Fourier domain and predicts the missing frequencies to learn visual representations for ViT and CNN without mask tokens, achieving competitive performance and robustness against prior MIM methods.

ABSTRACT

We present Masked Frequency Modeling (MFM), a unified frequency-domain-based approach for self-supervised pre-training of visual models. Instead of randomly inserting mask tokens to the input embeddings in the spatial domain, in this paper, we shift the perspective to the frequency domain. Specifically, MFM first masks out a portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum. Our key insight is that predicting masked components in the frequency domain is more ideal to reveal underlying image patterns rather than predicting masked patches in the spatial domain, due to the heavy spatial redundancy. Our findings suggest that with the right configuration of mask-and-predict strategy, both the structural information within high-frequency components and the low-level statistics among low-frequency counterparts are useful in learning good representations. For the first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token. Experimental results on image classification and semantic segmentation, as well as several robustness benchmarks show the competitive performance and advanced robustness of MFM compared with recent masked image modeling approaches. Furthermore, we also comprehensively investigate the effectiveness of classical image restoration tasks for representation learning from a unified frequency perspective and reveal their intriguing relations with our MFM approach.

Motivation & Objective

• Investigate whether masking in the frequency domain can yield better self-supervised representations than spatial masking.
• Develop a flexible, architecture-agnostic pre-training framework (ViT and CNN) that does not rely on mask tokens.
• Compare frequency-domain corruption with traditional low-level spatial corruptions and existing masked image modeling (MIM) methods.
• Evaluate MFM on image classification and semantic segmentation, and assess robustness across several benchmarks.
• Explore relationships between classical image restoration tasks and MFM from a unified frequency perspective.

Proposed method

• Convert images to frequency domain using FFT and mask a portion of frequency components via a low-/high-pass filter with a circular mask of radius r.
• Randomly select between low-pass and high-pass masked inputs and feed the corrupted spatial images to the encoder (ViT or CNN) without inserting mask tokens.
• Use a lightweight linear decoder to reconstruct the masked frequencies on the frequency spectrum via a frequency-domain loss.
• Define the reconstruction loss as a frequency distance that combines magnitude and phase differences across the masked spectrum (L = average over masked spectrum of |F_r - F_o|^gamma, gamma typically 1).
• Train on ImageNet-1K in a self-supervised manner and evaluate downstream on ImageNet-1K fine-tuning and ADE20K semantic segmentation.
• Show that predicting only the masked spectrum is more effective than reconstructing the full spectrum, and that a frequency-domain loss outperforms spatial losses.

Experimental results

Research questions

• RQ1 Does masking in the frequency domain enable learning richer representations for ViT and CNN without mask tokens?
• RQ2 How do mask type (low-/high-pass/random), radius, shape, and sampling affect MFM performance?
• RQ3 How does MFM compare to low-level image restoration tasks and to existing MIM methods in terms of performance and robustness?
• RQ4 Can MFM achieve competitive results on ImageNet classification and ADE20K segmentation with architectures like ViT and ResNet-50?
• RQ5 What is the impact of MFM on robustness to adversarial and common corruptions across benchmarks?

Key findings

• MFM achieves 83.1% top-1 on ViT-B/16 and 81.6% on ViT-S/16 after 300-epoch pre-training on ImageNet-1K (no mask tokens).
• On ADE20K, ViT-B/16 with MFM reaches 48.6 mIoU, surpassing several self-supervised methods and the supervised baseline in some settings.
• MFM often ranks among the top methods for robustness benchmarks, while maintaining strong standard accuracy (e.g., robustness metrics in Table 6).
• Low-/high-pass frequency masking, random masking, and predicting only the masked spectrum contribute to improved performance over reconstructing the full spectrum.
• Compared with low-level image processing tasks (SR, deblurring, denoising), the frequency-domain perspective reveals their effectiveness and interactions with architecture (ViT vs CNN).