[Paper Review] EmMixformer: Mix transformer for eye movement recognition

Huafeng Qin, Hongyu Zhu+4 more|arXiv (Cornell University)|Jan 10, 2024

Gaze Tracking and Assistive Technology4 citations

TL;DR

EmMixformer proposes a novel hybrid transformer architecture that integrates Attention LSTM, standard Transformer, and Fourier Transformer modules to jointly model local temporal dependencies, long-range sequential patterns, and global frequency-domain features in eye movement data. The model achieves state-of-the-art performance with a verification error rate of 0.0795% on the GazeBase RAN dataset, significantly outperforming existing methods.

ABSTRACT

Eye movement (EM) is a new highly secure biometric behavioral modality that has received increasing attention in recent years. Although deep neural networks, such as convolutional neural network (CNN), have recently achieved promising performance, current solutions fail to capture local and global temporal dependencies within eye movement data. To overcome this problem, we propose in this paper a mixed transformer termed EmMixformer to extract time and frequency domain information for eye movement recognition. To this end, we propose a mixed block consisting of three modules, transformer, attention Long short-term memory (attention LSTM), and Fourier transformer. We are the first to attempt leveraging transformer to learn long temporal dependencies within eye movement. Second, we incorporate the attention mechanism into LSTM to propose attention LSTM with the aim to learn short temporal dependencies. Third, we perform self attention in the frequency domain to learn global features. As the three modules provide complementary feature representations in terms of local and global dependencies, the proposed EmMixformer is capable of improving recognition accuracy. The experimental results on our eye movement dataset and two public eye movement datasets show that the proposed EmMixformer outperforms the state of the art by achieving the lowest verification error.

Motivation & Objective

• To address the limitation of existing deep learning models in capturing both local and global temporal dependencies in eye movement sequences.
• To enhance eye movement biometric recognition accuracy by integrating multiple attention mechanisms across time and frequency domains.
• To develop a unified end-to-end deep learning framework that effectively fuses short-term, long-term, and global contextual representations in eye movement data.
• To establish a new state-of-the-art benchmark in eye movement recognition using a novel mixed transformer architecture.

Proposed method

• The proposed EmMixformer employs a mixed block architecture combining three components: standard Transformer for long-range sequence modeling, Attention LSTM for learning short-term temporal dependencies, and Fourier Transformer for capturing global frequency-domain patterns.
• The Attention LSTM module enhances standard LSTM by incorporating a self-attention mechanism to better focus on relevant time steps and improve feature representation in short sequences.
• The Fourier Transformer applies self-attention in the frequency domain after transforming the input signal via the Fast Fourier Transform (FFT), enabling the model to learn global structural patterns.
• The three modules are concatenated and processed in parallel within the mixed block, allowing complementary learning of local, long-range, and global features.
• The model is trained end-to-end using cross-entropy loss on eye movement sequences, with verification error rate (EER) used as the primary evaluation metric.
• Ablation studies are conducted by progressively adding components (Transformer → LSTM+Transformer → attLSTM+Transformer → full EmMixFormer) to validate the contribution of each module.

Experimental results

Research questions

• RQ1 Can a hybrid transformer architecture that combines time-domain and frequency-domain attention mechanisms improve eye movement recognition accuracy compared to standard models?
• RQ2 To what extent do Attention LSTM and Fourier Transformer contribute to modeling short-term and global dependencies in eye movement sequences?
• RQ3 How does the integration of multiple attention mechanisms (self-attention in time, attention-enhanced LSTM, and frequency-domain attention) affect the model's generalization and robustness?
• RQ4 Does the proposed EmMixFormer architecture achieve state-of-the-art performance across diverse eye movement datasets, including those with varying temporal sampling intervals?

Key findings

• EmMixFormer achieves a verification error rate (EER) of 0.0795% on the RAN sub-dataset of GazeBase, representing a significant improvement over the baseline Transformer (0.1220% EER).
• The ablation study confirms that adding Attention LSTM to the Transformer reduces EER from 0.1220% to 0.0897%, demonstrating the value of modeling short-term dependencies.
• Incorporating the Fourier Transformer into the attLSTM+Transformer configuration further reduces EER to 0.0795%, confirming the importance of global frequency-domain feature learning.
• The performance degradation observed in all models with increasing time interval duration highlights the challenge of temporal variability, which EmMixFormer mitigates effectively through multi-scale attention.
• The model outperforms existing state-of-the-art methods on three datasets, including the authors’ own eye movement dataset and two public benchmarks, establishing a new SOTA in eye movement recognition.
• The ablation results show that the attention mechanism in LSTM (attLSTM) improves performance over standard LSTM, and the combination of all three modules yields the optimal recognition accuracy.