[Paper Review] MIDAS: Deep learning human action intention prediction from natural eye movement patterns

Paul Festor, Ali Shafti+4 more|arXiv (Cornell University)|Jan 22, 2022

Gaze Tracking and Assistive Technology4 citations

TL;DR

This paper proposes MIDAS, a deep learning model that predicts human action intention from natural eye movement patterns using only gaze and object position data. By modeling gaze dynamics as a time series with a bidirectional LSTM, MIDAS achieves 91.9% accuracy in classifying manipulation vs. inspection intentions without requiring gaze gestures or dwell time, demonstrating the feasibility of natural gaze as a Zero-UI interface.

ABSTRACT

Eye movements have long been studied as a window into the attentional mechanisms of the human brain and made accessible as novelty style human-machine interfaces. However, not everything that we gaze upon, is something we want to interact with; this is known as the Midas Touch problem for gaze interfaces. To overcome the Midas Touch problem, present interfaces tend not to rely on natural gaze cues, but rather use dwell time or gaze gestures. Here we present an entirely data-driven approach to decode human intention for object manipulation tasks based solely on natural gaze cues. We run data collection experiments where 16 participants are given manipulation and inspection tasks to be performed on various objects on a table in front of them. The subjects' eye movements are recorded using wearable eye-trackers allowing the participants to freely move their head and gaze upon the scene. We use our Semantic Fovea, a convolutional neural network model to obtain the objects in the scene and their relation to gaze traces at every frame. We then evaluate the data and examine several ways to model the classification task for intention prediction. Our evaluation shows that intention prediction is not a naive result of the data, but rather relies on non-linear temporal processing of gaze cues. We model the task as a time series classification problem and design a bidirectional Long-Short-Term-Memory (LSTM) network architecture to decode intentions. Our results show that we can decode human intention of motion purely from natural gaze cues and object relative position, with $91.9\%$ accuracy. Our work demonstrates the feasibility of natural gaze as a Zero-UI interface for human-machine interaction, i.e., users will only need to act naturally, and do not need to interact with the interface itself or deviate from their natural eye movement patterns.

Motivation & Objective

• To overcome the Midas Touch problem in gaze-based human-machine interfaces by decoding intention from natural gaze cues without requiring deliberate gaze gestures or dwell time.
• To investigate whether natural eye movement patterns alone—without artificial task constraints—can reliably predict whether a person intends to manipulate or inspect an object.
• To develop a data-driven, end-to-end deep learning framework that leverages temporal dynamics of gaze and object relationships for intention recognition.
• To validate that intention prediction relies on non-linear temporal processing of gaze, not just spatial fixation patterns or simple heuristics.
• To establish a Zero-UI interaction paradigm where users interact naturally without modifying their gaze behavior or interface interactions.

Proposed method

• Collected ego-centric eye-tracking data from 16 participants performing manipulation and inspection tasks using wearable eye-trackers in a natural, head-free environment.
• Used the Semantic Fovea model to detect and localize objects in the scene and compute their spatial relationship to gaze positions at each frame.
• Interpolated object position data (up to 100 points, ~0.8s gaps) to align with the higher-resolution gaze data (~120Hz), distinguishing between detection failures and object absence.
• Cropped all trials to a uniform 3.33-second window post-instruction to eliminate bias from trial length differences.
• Designed a bidirectional LSTM-based architecture (2x256 LSTM layers, attention, 3 dense layers with ReLU and sigmoid activations) to model temporal dependencies in gaze-object sequences.
• Trained the model using binary cross-entropy loss with Adam optimizer (initial LR: 2.5×10⁻⁴) over 200 epochs, with k-fold and leave-one-subject-out cross-validation for robustness.

Experimental results

Research questions

• RQ1 Can human action intention (manipulation vs. inspection) be accurately predicted from natural, unconstrained gaze patterns without requiring gaze gestures or dwell time?
• RQ2 Does intention prediction rely on non-linear temporal processing of gaze dynamics, or is it reducible to simple spatial fixation patterns?
• RQ3 To what extent do confounding factors such as task description length or number of named objects bias intention prediction models?
• RQ4 Can a deep learning model trained on raw gaze and object position sequences achieve high accuracy in intention decoding without explicit handcrafted features?
• RQ5 Is it feasible to build a Zero-UI interface where users interact naturally, and systems infer intention purely from gaze behavior?

Key findings

• MIDAS achieved 91.9% accuracy in classifying human action intention (manipulation vs. inspection) using only natural gaze traces and object positions.
• The model's performance was significantly better than baseline classifiers trained only on distance features, indicating that intention prediction relies on complex temporal dynamics, not just spatial proximity.
• Bias from longer task descriptions was mitigated by cropping trials to a uniform 3.33-second window after the audio instruction ended, ensuring no length-based cues influenced predictions.
• The model demonstrated robustness across subjects, as confirmed by leave-one-subject-out cross-validation, indicating generalization to unseen users.
• The study shows that non-linear temporal modeling of gaze sequences is essential for intention decoding, as simple spatial or static fixation-based models performed poorly.
• The results validate the feasibility of using natural gaze as a Zero-UI interface, where users need not adapt their gaze behavior to interact with systems.