[论文解读] Automatic Gaze Analysis: A Survey of Deep Learning based Approaches

Shreya Ghosh, Abhinav Dhall+3 more|arXiv (Cornell University)|Aug 12, 2021

Gaze Tracking and Assistive Technology参考文献 199被引用 16

一句话总结

本综述全面回顾了基于深度学习的自动眼动分析方法，重点关注无监督、自监督和弱监督方法。它识别了非受限环境中的关键挑战，并为增强现实/虚拟现实（AR/VR）、人机交互（HCI）及计算机视觉应用中的鲁棒、实时眼动估计提出了未来研究方向。

ABSTRACT

Eye gaze analysis is an important research problem in the field of Computer Vision and Human-Computer Interaction. Even with notable progress in the last 10 years, automatic gaze analysis still remains challenging due to the uniqueness of eye appearance, eye-head interplay, occlusion, image quality, and illumination conditions. There are several open questions, including what are the important cues to interpret gaze direction in an unconstrained environment without prior knowledge and how to encode them in real-time. We review the progress across a range of gaze analysis tasks and applications to elucidate these fundamental questions, identify effective methods in gaze analysis, and provide possible future directions. We analyze recent gaze estimation and segmentation methods, especially in the unsupervised and weakly supervised domain, based on their advantages and reported evaluation metrics. Our analysis shows that the development of a robust and generic gaze analysis method still needs to address real-world challenges such as unconstrained setup and learning with less supervision. We conclude by discussing future research directions for designing a real-world gaze analysis system that can propagate to other domains including Computer Vision, Augmented Reality (AR), Virtual Reality (VR), and Human Computer Interaction (HCI). Project Page: https://github.com/i-am-shreya/EyeGazeSurvey}{https://github.com/i-am-shreya/EyeGazeSurvey

研究动机与目标

• 分析基于深度学习的眼动估计与分割的最新进展，尤其关注弱监督和无监督设置下的方法。
• 识别在现实世界非受限条件下、监督信息有限时有效的眼动分析技术。
• 利用针对眼动分析量身定制的领域特定指标与基准协议，评估现有方法。
• 探索在AR/VR和人机交互中实现鲁棒、低延迟眼动估计的未来研究方向。
• 通过提出混合学习框架，弥合传统基于模型的方法与数据驱动的深度学习之间的差距。

提出的方法

• 系统性回顾100余篇近期关于眼动估计、分割与跟踪的深度学习相关研究。
• 根据监督程度对方法进行分类：完全监督、弱监督、自监督与无监督。
• 分析关键组件：眼部检测（配准）、特征表示（如CNNs、ViTs）以及眼动预测（回归或分类）。
• 使用标准基准（如CAVE、MPII、ETH-XGaze和TabletGaze）及指标（如平均绝对误差（MAE）和角度误差）评估方法性能。
• 调查数据采集设备，包括RGB/IR摄像头、笔记本/网络摄像头以及专用眼动仪（如视频眼动测量法）。
• 整合计算机视觉、AR/VR与HCI领域的洞见，提出结合几何眼动模型与深度外观特征的混合模型。

Figure 1: A brief chronology of seminal gaze analysis works. The very first gaze pattern modelling dates back to the work of Javal et al. in 1879 [ 4 ] . One of the first deep learning driven appearance based gaze estimation models was proposed in $2015$ [ 24 ] .

实验结果

研究问题

• RQ1 在非受限环境中，哪些深度学习架构与训练范式在眼动估计中最为有效？
• RQ2 与完全监督方法相比，无监督与弱监督方法在准确率与泛化能力方面表现如何？
• RQ3 现实世界眼动估计中的主要挑战是什么？当前方法如何应对这些挑战？
• RQ4 如何利用眼动推理来推断认知与情感状态，而不仅限于简单的注视方向？
• RQ5 多模态或跨模态输入（如音频、头部姿态）在提升眼动估计鲁棒性方面发挥何种作用？

主要发现

• 无监督与自监督方法在减少对昂贵且易出错的人工标注眼动标签依赖方面展现出巨大潜力。
• 当前最先进模型在ETH-XGaze等基准数据集上，在受控条件下可实现低于1.5度的平均绝对误差。
• 结合几何眼动模型与深度外观特征的混合模型可显著提升在不同头部姿态与光照条件下的泛化能力。
• 未来的眼动轨迹预测正成为AR/VR应用中实现低延迟视网膜聚焦渲染的关键推动因素。
• 利用音频或头部运动线索的多模态方法可在低可见度或遮挡场景中提升眼动估计性能。
• 尽管已取得进展，但在非受限真实环境（尤其是极端头部姿态与遮挡情况）下的鲁棒性仍是主要开放挑战。

Figure 2: Top Left: Overview of the human visual system, eye modelling and eye movement. For computer vision based automated gaze analysis, we consider an image containing eyes (left) as input. Thus, such methods analyze the visible eye regions (middle) and predict the 2-D/3-D gaze vector as output.