[Paper Review] Peripheral Vision Transformer

Juhong Min, Yucheng Zhao+2 more|arXiv (Cornell University)|Jun 14, 2022

CCD and CMOS Imaging Sensors18 citations

TL;DR

This paper proposes PerViT, a vision transformer that injects peripheral inductive biases via a novel Multi-head Peripheral Attention (MPA) mechanism, enabling the network to learn hierarchical visual perception similar to human peripheral vision. By modeling attention across torus-shaped regions based on gaze distance, PerViT achieves state-of-the-art accuracy on ImageNet-1K across multiple model sizes, with improvements of up to 4.2% top-1 accuracy over DeiT baselines.

ABSTRACT

Human vision possesses a special type of visual processing systems called peripheral vision. Partitioning the entire visual field into multiple contour regions based on the distance to the center of our gaze, the peripheral vision provides us the ability to perceive various visual features at different regions. In this work, we take a biologically inspired approach and explore to model peripheral vision in deep neural networks for visual recognition. We propose to incorporate peripheral position encoding to the multi-head self-attention layers to let the network learn to partition the visual field into diverse peripheral regions given training data. We evaluate the proposed network, dubbed PerViT, on ImageNet-1K and systematically investigate the inner workings of the model for machine perception, showing that the network learns to perceive visual data similarly to the way that human vision does. The performance improvements in image classification over the baselines across different model sizes demonstrate the efficacy of the proposed method.

Motivation & Objective

• To bridge the gap between human and machine vision by incorporating biological principles of peripheral vision into deep neural networks.
• To address the limitations of standard self-attention in modeling spatial hierarchies by introducing region-specific attention based on distance from the gaze center.
• To improve visual recognition performance by enabling transformers to learn fine-grained local features and coarse global context simultaneously, like the human visual system.
• To validate that the proposed mechanism leads to attention patterns that resemble human visual processing without requiring explicit supervision.
• To demonstrate the effectiveness of peripheral inductive biases across different model sizes and downstream tasks.

Proposed method

• The method introduces a peripheral position encoding (Φp) that assigns positional embeddings based on the distance from the center of the image, modeling torus-shaped attention regions.
• A Multi-head Peripheral Attention (MPA) layer is proposed, where queries and keys are projected into peripheral regions using a learnable projection head with a 3×3 local receptive field.
• The MPA mechanism uses normalized coordinates ([-1,1]) to stabilize training and ensures spatial consistency in attention computation.
• The model uses a hybrid architecture with four stages, each with increasing channel dimensions and attention heads, to capture multi-scale features.
• The peripheral projection head has a dimension of 4×N_h, and the neighborhood size K=3 is used to capture local spatial context in the peripheral representation.
• Training follows DeiT’s recipe with data augmentation, label smoothing, and mixup, using AdamW optimizer and cosine learning rate decay.

Experimental results

Research questions

• RQ1 Can injecting peripheral inductive biases into self-attention layers improve visual recognition performance in vision transformers?
• RQ2 Do the attention maps in the proposed PerViT model exhibit hierarchical patterns resembling human peripheral vision, with central regions focusing on fine details and peripheral regions on global context?
• RQ3 How does the proposed peripheral position encoding compare to standard learnable position embeddings in terms of accuracy and efficiency?
• RQ4 Does the model generalize well to transfer learning tasks such as CIFAR-100 and iNaturalist-19?
• RQ5 Can the peripheral attention mechanism be effective across different model sizes without requiring architectural overhaul?

Key findings

• PerViT achieves up to 4.2% higher top-1 accuracy on ImageNet-1K compared to DeiT baselines, with the largest improvement seen in the Tiny model.
• The peripheral position encoding (Φp) contributes significantly to performance, with a 1.4% to 4.2% accuracy gain across model sizes, despite being less than 0.6% of the total model size.
• The model shows strong generalization, surpassing DeiT-B by approximately 1%p on CIFAR-100 and iNaturalist-19, while matching it on CIFAR-10.
• Qualitative analysis of attention maps reveals that early layers attend to central regions for fine-grained details, while deeper layers attend globally to peripheral regions, mimicking human visual processing.
• The method is efficient, with the peripheral encoding adding minimal computational cost (0.31M parameters in PerViT-M) and maintaining low FLOPs.
• The model’s performance is robust across different training recipes and data augmentations, including RandAugment and mixup, indicating strong generalization.