[논문 리뷰] SpectFormer: Frequency and Attention is what you need in a Vision Transformer

Badri N. Patro, Vinay P. Namboodiri외 1명|arXiv (Cornell University)|2023. 04. 13.

Advanced Neural Network Applications인용 수 53

한 줄 요약

SpectFormer는 Fourier 기반 스펙트럴 계층과 후기 단계의 다중 헤드 자기 주의를 결합하여 Vision Transformer의 성능을 향상시키고, 작은/기본 변형에서 ImageNet-1K에 대한 최첨단 결과를 달성하며 전이/COCO에서도 강력한 성능을 보입니다.

ABSTRACT

Vision transformers have been applied successfully for image recognition tasks. There have been either multi-headed self-attention based (ViT \cite{dosovitskiy2020image}, DeIT, \cite{touvron2021training}) similar to the original work in textual models or more recently based on spectral layers (Fnet\cite{lee2021fnet}, GFNet\cite{rao2021global}, AFNO\cite{guibas2021efficient}). We hypothesize that both spectral and multi-headed attention plays a major role. We investigate this hypothesis through this work and observe that indeed combining spectral and multi-headed attention layers provides a better transformer architecture. We thus propose the novel Spectformer architecture for transformers that combines spectral and multi-headed attention layers. We believe that the resulting representation allows the transformer to capture the feature representation appropriately and it yields improved performance over other transformer representations. For instance, it improves the top-1 accuracy by 2\% on ImageNet compared to both GFNet-H and LiT. SpectFormer-S reaches 84.25\% top-1 accuracy on ImageNet-1K (state of the art for small version). Further, Spectformer-L achieves 85.7\% that is the state of the art for the comparable base version of the transformers. We further ensure that we obtain reasonable results in other scenarios such as transfer learning on standard datasets such as CIFAR-10, CIFAR-100, Oxford-IIIT-flower, and Standford Car datasets. We then investigate its use in downstream tasks such of object detection and instance segmentation on the MS-COCO dataset and observe that Spectformer shows consistent performance that is comparable to the best backbones and can be further optimized and improved. Hence, we believe that combined spectral and attention layers are what are needed for vision transformers.

연구 동기 및 목표

• 이미지 표현을 위해 스펙트럴 토큰 혼합과 주의 기반 토큰 혼합을 결합해야 할 필요성을 제시한다.
• 스펙트럴 레이어를 초기에 사용하고 주의 레이어를 나중에 두는 통합 SpectFormer 아키텍처를 설계한다.
• ImageNet 및 다운스트림 태스크에서 일반(본래) 트랜스포머, 스펙트럴 트랜스포머, 계층적 트랜스포머와 비교하여 SpectFormer를 실증적으로 검증한다.
• SpectFormer의 전이 학습 및 객체 탐지/세분화 성능 이점을 입증한다.

제안 방법

• SpectFormer를 두 부분으로 구성된 트랜스포머 블록으로 도입한다: 로컬 주파수 캡처를 위한 스펙트럴 레이어(FFT 기반 게이팅)와 그 뒤의 글로벌 특징을 위한 다중 헤드 자기 주의.
• 블록에서 스펙트럴 레이어와 주의 레이어의 비율을 제어하는 조정 가능한 alpha를 사용한다.
• 표준 ViT-스타일 파이프라인에서 패치 임베딩 계층, 위치 인코딩 및 분류 헤드를 활용한다.
• 여러 스펙트럴 변형(FN, FGN, FNO, WGN)을 실험하고 Fourier Gating Network(FGN)가 가장 효과적임을 확인한다.
• 원래 SpectFormer와 계층형 SpectFormer 변형을 DeIT, GFNet, AFNO, LiT, Swin, PVT와 비교하여 ImageNet-1K 및 전이 데이터 세트에서 평가한다.

실험 결과

연구 질문

• RQ1 mixes spectral layers with multi-headed attention outperform purely spectral or purely attention-based transformers for vision tasks?
• RQ2 What is the optimal allocation (alpha) between spectral and attention layers for best ImageNet performance?
• RQ3 How does SpectFormer perform in transfer learning scenarios (CIFAR, Flowers, Cars) compared to baselines?
• RQ4 Are SpectFormer variants effective for downstream tasks like object detection and instance segmentation on MS COCO?

주요 결과

• SpectFormer-S achieves 84.25% top-1 accuracy on ImageNet-1K for the small variant.
• SpectFormer-L achieves 85.7% top-1 accuracy on ImageNet-1K for the base-like large variant.
• SpectFormer outperforms GFNet, AFNO, LiT, and DeiT across sizes, with hierarchical variants delivering state-of-the-art results.
• Fourier Gating Network (FGN) as the spectral block variant yields the best ablation results among FN, FNO, WGN, and FGN.
• SpectFormer shows consistent transfer learning gains on CIFAR-10, CIFAR-100, Flowers, and Cars, and competitive performance on MS COCO object detection/segmentation.