[Paper Review] Real-time Pupil Tracking from Monocular Video for Digital Puppetry

Artsiom Ablavatski, Andrey Vakunov+3 more|arXiv (Cornell University)|Jun 19, 2020

Video Analysis and Summarization12 references18 citations

TL;DR

This paper presents a real-time, monocular video-based pupil tracking system for digital puppetry on mobile devices, using a lightweight neural network to detect 5 key eye landmarks (pupil center and outer iris) and a displacement-based heuristic to estimate pupil blend shape coefficients. The method achieves over 50 FPS on modern phones and enables lifelike, responsive eye animation in virtual avatars without calibration or external sensors.

ABSTRACT

We present a simple, real-time approach for pupil tracking from live video on mobile devices. Our method extends a state-of-the-art face mesh detector with two new components: a tiny neural network that predicts positions of the pupils in 2D, and a displacement-based estimation of the pupil blend shape coefficients. Our technique can be used to accurately control the pupil movements of a virtual puppet, and lends liveliness and energy to it. The proposed approach runs at over 50 FPS on modern phones, and enables its usage in any real-time puppeteering pipeline.

Motivation & Objective

• To enable real-time, accurate pupil tracking in monocular video for digital puppeteering on mobile devices.
• To address the lack of expressivity in virtual avatars due to static or absent pupil movement.
• To develop a calibration-free, end-to-end pipeline that integrates with existing face mesh models.
• To achieve real-time performance (50+ FPS) on consumer mobile hardware using lightweight models.
• To enable natural, dynamic eye movements in virtual puppets through blend shape estimation from detected landmarks.

Proposed method

• A small, efficient neural network processes cropped 64×64 eye regions from a face mesh prediction to detect 5 key 2D eye landmarks: pupil center and 4 outer iris points.
• The detected landmarks are fused with the original face mesh by replacing x,y coordinates while preserving z-depth, extending the mesh to 478 vertices.
• A displacement-based method computes blend shape coefficients by comparing current vertex distances to empirically derived neutral and activated states for four pupil directions (inward, outward, up, down).
• A real-time standard score filter calibrates the displacement thresholds dynamically using a circular buffer of trusted measurements to adapt to individual user variations.
• Post-processing includes smoothing and inter-eye coupling to reduce jitter and ensure visually appealing, stable eye animation.
• The pipeline runs efficiently on mobile devices using TensorFlow Lite with GPU acceleration via MediaPipe.

Experimental results

Research questions

• RQ1 Can accurate pupil tracking be achieved in real time from monocular video on mobile devices without external sensors or calibration?
• RQ2 How can pupil movement be mapped to meaningful blend shape coefficients for use in virtual puppet animation?
• RQ3 What is the optimal balance between model accuracy, inference speed, and memory footprint for mobile deployment?
• RQ4 How can person-specific variations in eye geometry be compensated for in real time without pre-calibration?
• RQ5 Can a lightweight, end-to-end system be built that integrates seamlessly with existing face mesh pipelines for real-time puppeteering?

Key findings

• The model achieved a Mean Absolute Distance normalized by Inter-Eye Distance (MAD IED) of 7.16% on a held-out test set, outperforming baseline manual annotation error (7.04% for hard cases).
• The system runs at over 50 FPS on modern smartphones, with inference times as low as 2.6ms on the iPhone X for the eye refinement network.
• The real-time standard score filter successfully adapted to individual user variations, stabilizing displacement thresholds over time as shown in Figure 3.
• The method enables smooth, jitter-free eye animation through post-processing and inter-eye coupling, enhancing visual realism.
• The approach generalizes to any face mesh pipeline and requires no person-specific calibration or depth sensors.
• The final pipeline supports real-time puppeteering with lifelike pupil dynamics, as demonstrated in Figure 1 with rendered virtual avatars.