[논문 리뷰] Focal Self-attention for Local-Global Interactions in Vision Transformers

Jianwei Yang, Chunyuan Li외 5명|arXiv (Cornell University)|2021. 07. 01.

Visual Attention and Saliency Detection참고 문헌 87인용 수 267

한 줄 요약

이 논문은 비전 트랜스포머 내에서 미세한 로컬 상호작용과 거시적 글로벌 상호작용을 결합하는 focal self-attention을 소개하여 다양한 모델 규모에서 ImageNet, COCO, ADE20K에서 최첨단 결과를 도출한다.

ABSTRACT

Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through self-attention is arguably the main source for the success. But it also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). In this paper, we present focal self-attention, a new mechanism that incorporates both fine-grained local and coarse-grained global interactions. Using this new mechanism, each token attends the closest surrounding tokens at fine granularity but the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a larger size of 89.8M achieve 83.5 and 83.8 Top-1 accuracy, respectively, on ImageNet classification at 224x224 resolution. Using Focal Transformers as the backbones, we obtain consistent and substantial improvements over the current state-of-the-art Swin Transformers for 6 different object detection methods trained with standard 1x and 3x schedules. Our largest Focal Transformer yields 58.7/58.9 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks.

연구 동기 및 목표

• 고해상도 비전 작업(예: 탐지 및 분할)에 대한 전체 self-attention의 이차 비용을 동기 부여하고 해결한다.
• 로컬 미세한 상호작용과 글로벌 거친 상호작용을 효율적으로 모델링하기 위한 focal self-attention를 제안한다.
• 정확한 밀도 예측을 위한 다중 스케일 아키텍처를 갖춘 Focal Transformer 변형들을 개발한다.
• 분류, 탐지, 및 분할 작업에서 SoTA 변환기 대비 개선을 경험적으로 검증한다.

제안 방법

• 근접 토큰에 대해 미세한 입자 수준으로, 원거리 토큰에 대해 거친 입자 수준으로 주의를 기울이는 focal self-attention을 정의한다.
• 특징 맵을 창(window)으로 분할하고 다중 focal 레벨을 위해 서브 윈도우를 풀링하여 윈도우 기반 focal self-attention을 구현한다.
• 선형 프로젝션으로 쿼리, 키, 값을 계산하고 상대 위치 바이어스로 다단계 주의를 적용한다.
• 고해상도 입력을 처리하기 위해 패치 임베딩과 계(stage)별 focal 블록을 갖춘 다단계, 다중 스케일 아키텍처를 채택한다.
• ImageNet-1K, COCO, 및 ADE20K에서 Focal Transformer 변형(Focal-Tiny, Focal-Small, Focal-Base)을 학습 및 평가하고 Swin Transformers 및 다른 기준선과 비교한다.

실험 결과

연구 질문

• RQ1 Can focal self-attention capture both local and global interactions in Vision Transformers without incurring quadratic computational cost?
• RQ2 Do multi-scale, window-based focal mechanisms improve performance on image classification, object detection, and semantic segmentation compared to existing attention strategies?
• RQ3 How do Focal Transformer variants perform relative to state-of-the-art models across standard benchmarks?

주요 결과

• Focal Transformers outperform SoTA Vision Transformer baselines of similar size and FLOPs on ImageNet-1K classification.
• Focal-Small and Focal-Base achieve higher Top-1 accuracy than comparable Swin and other Transformer models.
• In object detection and instance segmentation on COCO, Focal-Tiny/Small/Base provide consistent gains over Swin Transformers across multiple detectors and schedules.
• On ADE20K semantic segmentation, Focal-Tiny/Small/Base outperform Swin Transformers of similar size in single-scale and multi-scale settings.
• The proposed attention mechanism enables both short-range fine-grain and long-range coarse-grain interactions with reduced computational cost compared to full attention.