[Paper Review] Measuring eye-tracking accuracy and its impact on usability in apple vision pro

Zehao Huang, Gancheng Zhu+5 more|arXiv (Cornell University)|Jun 1, 2024

Leaf Properties and Growth Measurement5 citations

TL;DR

The study evaluates Apple Vision Pro’s eye-tracking accuracy in two test setups and examines its relation to usability, finding average accuracies of 1.11° and 0.93° within roughly 34°×18° FOV, SUS usability scores of 75.24 and 68.26, and no reliable correlation between eye-tracking accuracy and usability.

ABSTRACT

With built-in eye-tracking cameras, the recently released Apple Vision Pro (AVP) mixed reality (MR) headset features gaze-based interaction, eye image rendering on external screens, and iris recognition for device unlocking. One of the technological advancements of the AVP is its heavy reliance on gaze- and gesture-based interaction. However, limited information is available regarding the technological specifications of the eye-tracking capability of the AVP, and raw gaze data is inaccessible to developers. This study evaluates the eye-tracking accuracy of the AVP with two sets of tests spanning both MR and virtual reality (VR) applications. This study also examines how eye-tracking accuracy relates to user-reported usability. The results revealed an overall eye-tracking accuracy of 1.11° and 0.93° in two testing setups, within a field of view (FOV) of approximately 34° x 18°. The usability and learnability scores of the AVP, measured using the standard System Usability Scale (SUS), were 75.24 and 68.26, respectively. Importantly, no statistically reliable correlation was found between eye-tracking accuracy and usability scores. These results suggest that eye-tracking accuracy is critical for gaze-based interaction, but it is not the sole determinant of user experience in VR/AR.

Motivation & Objective

• Assess eye-tracking accuracy of Apple Vision Pro in MR/VR contexts.
• Quantify usability and learnability using SUS metrics.
• Investigate correlation between eye-tracking accuracy and usability outcomes.

Proposed method

• Conduct two sets of eye-tracking accuracy tests in MR and VR contexts.
• Measure gaze accuracy within a ~34°×18° field of view (FOV).
• Compute System Usability Scale (SUS) scores for usability and learnability.

Experimental results

Research questions

• RQ1 What is the eye-tracking accuracy of Apple Vision Pro in the tested MR/VR setups?
• RQ2 How usable and learnable is the Apple Vision Pro according to SUS?
• RQ3 Is there a statistically reliable relationship between eye-tracking accuracy and usability metrics?
• RQ4 Does eye-tracking accuracy fall within the device’s operational FOV during tasks?

Key findings

• Eye-tracking accuracy averaged 1.11° in one setup and 0.93° in another within ~34°×18° FOV.
• Usability and learnability scores (SUS) were 75.24 and 68.26 respectively.
• No statistically reliable correlation was found between eye-tracking accuracy and usability scores.
• Eye-tracking accuracy is important for gaze-based interaction but does not solely determine user experience.