[Paper Review] What Are You Looking at? Detecting Human Intention in Gaze based Human-Robot Interaction

Lei Shi, Cosmin Copot+1 more|arXiv (Cornell University)|Sep 17, 2019

Gaze Tracking and Assistive Technology25 references4 citations

TL;DR

This paper proposes a gaze-intention detection method for human-robot interaction that uses Earth Mover’s Distance (EMD) to compare hypothetical gaze distributions—derived from saliency maps of objects—against actual gaze data from eye-tracking glasses. Unlike fixation-based methods, it remains robust during saccades and gaze noise, achieving 92.2% success in object selection within successful predictions and reducing deletion errors by up to 75% compared to fixation-based approaches.

ABSTRACT

In gaze based Human-Robot Interaction (HRI), it is important to determine the human intention for further interaction. The gaze intention is often modelled as fixation. However, when looking at an object, it is not natural and it is difficult to maintain the gaze fixating on one point for a long time. Saccades may happen while a human is still focusing on the object. The prediction of human intention will be lost during saccades. In addition, while the human intention is on object, the gazes may be located outside of the object bounding box due to different noise sources, which would cause false negative predictions. In this work, we propose a novel approach to detect whether a human is focusing on an object in HRI application. We determine the gaze intention by comparing the similarity between the hypothetic gazes on objects and the actual gazes. We use Earth Mover's Distance (EMD) to measure the similarity and 1 Nearest Neighbour to classify which object a human is looking at. Our experimental results indicate that, compare to fixation, our method can successfully determine the human intention even during saccadic eye movements and increase the classification accuracy with noisy gaze data. We also demonstrate that, in the interaction with a robot, the proposed approach can obtain a high accuracy of object selection within successful predictions.

Motivation & Objective

• To address the limitations of fixation-based gaze intention detection in human-robot interaction, particularly during saccadic eye movements and due to gaze intention errors.
• To improve robustness against noise sources such as bounding box variation, calibration drift, and head movement in mobile eye-tracking systems.
• To develop a method that accurately detects human visual intention even when gaze points fall outside object bounding boxes.
• To enable reliable object selection in HRI scenarios using gaze, with a confirmation gesture for interaction.

Proposed method

• Hypothetical gaze distributions are generated by sampling salient regions within object bounding boxes using a bottom-up visual saliency model.
• Actual gaze distributions are collected from mobile eye-tracking glasses during human observation of objects.
• Earth Mover’s Distance (EMD) is used to compute the similarity between hypothetical and actual gaze distributions.
• A 1-Nearest Neighbor (1-NN) classifier with null rejection is applied to assign the most similar object to the current gaze pattern.
• A simple eye gesture—closing eyes for 0.5 seconds—is used to confirm object selection in robot interaction.
• The method is compared against fixation-based classification using KL divergence and Bhattacharyya distance as alternative similarity measures.

Experimental results

Research questions

• RQ1 Can gaze intention be reliably detected during saccadic eye movements when fixation-based methods fail?
• RQ2 How does gaze noise—caused by bounding box variation or calibration drift—affect the accuracy of object selection in HRI?
• RQ3 Can a saliency-based gaze distribution model improve robustness compared to fixation-based approaches in noisy gaze data?
• RQ4 To what extent does EMD outperform KL divergence and Bhattacharyya distance in measuring gaze distribution similarity?
• RQ5 What is the success rate of object selection in a real HRI scenario using the proposed method?

Key findings

• The proposed method reduced deletion events—caused by saccades and gaze intention errors—by up to 75% compared to fixation-based methods, with deletion rates of 3.0–6.5% versus 17.0–26.3%.
• The method achieved a 92.2% success rate in object selection within successful predictions during robot interaction, demonstrating high reliability in real-world HRI.
• EMD outperformed KL divergence and Bhattacharyya distance in correlation with Euclidean distances between gaze and hypothesis points, and avoided saturation issues.
• The method maintained high accuracy even when gaze points were located outside object bounding boxes due to noise, whereas fixation-based methods failed under such conditions.
• When gaze data was clean and within bounding boxes, fixation-based methods performed comparably, but the proposed method significantly outperformed them under noisy conditions.