[Paper Review] Pistol: Pupil Invisible Supportive Tool to extract Pupil, Iris, Eye Opening, Eye Movements, Pupil and Iris Gaze Vector, and 2D as well as 3D Gaze

Wolfgang Fuhl, Daniel Weber+1 more|arXiv (Cornell University)|Jan 18, 2022

Gaze Tracking and Assistive Technology26 citations

TL;DR

Pistol is a feature extraction and gaze estimation toolkit that processes Pupil Invisible data (offline) to extract pupil/iris geometries, eye opening, eye movements, markers, and 2D/3D gaze using Levenberg–Marquardt fitting and neural networks.

ABSTRACT

This paper describes a feature extraction and gaze estimation software, named extit{Pistol} that can be used with Pupil Invisible projects and other eye trackers in the future. In offline mode, our software extracts multiple features from the eye including, the pupil and iris ellipse, eye aperture, pupil vector, iris vector, eye movement types from pupil and iris velocities, marker detection, marker distance, 2D gaze estimation for the pupil center, iris center, pupil vector, and iris vector using Levenberg Marquart fitting and neural networks. The gaze signal is computed in 2D for each eye and each feature separately and for both eyes in 3D also for each feature separately. We hope this software helps other researchers to extract state-of-the-art features for their research out of their recordings. Link: https://es-cloud.cs.uni-tuebingen.de/d/8e2ab8c3fdd444e1a135/?p=%2FPISTOL&mode=list

Motivation & Objective

• Provide a free tool to extract a broad set of eye features from pupil-invisible recordings.
• Enable 2D gaze estimation per eye and 3D gaze estimation across both eyes using multiple optimization methods.
• Deliver robust detection of pupil/iris landmarks, eyelids, eye opening, and eye movements to support research and industry applications.
• Support compatibility with future eye trackers and integration with Pupil Invisible projects.

Proposed method

• Use small DNNs with tensor normalization and full distribution training to detect pupil, iris, and eyelid landmarks.
• Fit pupil/iris ellipses with OpenCV ellipse fitting and cubic splines for eyelids.
• Estimate eye opening via an optimization constrained orthogonally to the eye-corners vector.
• Compute eyeball radius and center using a neural network and derive optical vectors from center-to-center vectors.
• Perform eye movement classification with a neural network using angles between pupil and iris vectors and opening differences.
• Detect calibration markers with a coarse and a fine DNN detector to support depth and gaze calibration.

Experimental results

Research questions

• RQ1 How effectively can Pistol extract comprehensive eye features (pupil/iris ellipses, eyelids, eyeball, vectors) from pupil-invisible data?
• RQ2 What is the accuracy of 2D/3D gaze estimation using Levenberg–Marquardt fitting and neural networks across per-eye and combined-eye configurations?
• RQ3 How reliable is marker-based calibration for depth estimation and gaze computation across varying head/eye poses?
• RQ4 What are the runtime characteristics and hardware requirements for real-time offline processing on legacy GPUs?

Key findings

• Pistol achieves pupil center RMSE of about 0.93 px and iris landmark RMSE of about 1.12 px on annotated data, with pupil/iris eyelid mIoU around 0.84–0.91 (Table 3).
• 2D gaze estimation averages around 21.8–32.3 px depending on method and eye, with combined-eye 2D estimates around 18.5–22.9 px (Table 5).
• 3D gaze estimation average errors range from ~19.6–20.5 px per method (Table 5).
• Marker detection coarse stage yields 6.70 px average landmark accuracy with 2.34% false detections; fine detector reduces to 0.82 px accuracy with 0.001% false detections (Table 7).
• Pistol runs at ~17.21 ms per frame for feature detection on a GTX 1050Ti, with marker detection taking ~57 ms per frame (Table 8).
• Depth estimation using a KNN approach yields practical accuracy with most errors below 50 cm, though near-field depth remains more challenging (Figure 10).