[Paper Review] Human Attention Estimation for Natural Images: An Automatic Gaze Refinement Approach

Jinsoo Choi, Tae-Hyun Oh+1 more|arXiv (Cornell University)|Jan 12, 2016

Visual Attention and Saliency Detection47 references4 citations

TL;DR

This paper proposes an implicit, user-agnostic method for real-time human attention estimation in natural images using integrated gaze tracking and computational saliency models. By leveraging webcam-based gaze estimation with incremental GMM clustering and saliency refinement, the approach learns individual attention preferences without active user participation, achieving accurate personal saliency maps and enabling applications like personalized photo retrieval and tag suggestion.

ABSTRACT

Photo collections and its applications today attempt to reflect user interactions in various forms. Moreover, photo collections aim to capture the users' intention with minimum effort through applications capturing user intentions. Human interest regions in an image carry powerful information about the user's behavior and can be used in many photo applications. Research on human visual attention has been conducted in the form of gaze tracking and computational saliency models in the computer vision community, and has shown considerable progress. This paper presents an integration between implicit gaze estimation and computational saliency model to effectively estimate human attention regions in images on the fly. Furthermore, our method estimates human attention via implicit calibration and incremental model updating without any active participation from the user. We also present extensive analysis and possible applications for personal photo collections.

Motivation & Objective

• To estimate human attention regions in natural images without requiring active user participation or specialized hardware.
• To integrate implicit gaze estimation with computational saliency models to refine attention maps based on individual viewing behavior.
• To enable real-time, incremental learning of user-specific attention preferences through passive observation of photo viewing.
• To demonstrate practical applications in personal photo collections, including saliency refinement, tag suggestion, and attention-based photo retrieval.
• To develop a scalable, off-the-shelf solution using standard webcams and minimal calibration for real-world deployment.

Proposed method

• Uses a webcam to capture eye gaze positions during natural photo viewing, enabling implicit calibration without explicit user input.
• Applies incremental Gaussian Mixture Model (GMM) clustering to learn user-specific gaze distribution patterns over time.
• Integrates the learned gaze distributions with a pre-trained computational saliency model via mode-seeking fusion to refine attention maps.
• Employs an efficient incremental learning pipeline that updates the GMM and Gaussian Process Regression (GPR) models in real time.
• Uses a similarity metric based on normalized saliency map overlap (Sim(s,t) = ∫ min(s(x), t(x)) dx) for photo retrieval tasks.
• Stores and updates GMM parameters (mean, covariance, weights, and count estimates) to reduce computational cost during online updates.

Experimental results

Research questions

• RQ1 Can implicit gaze estimation from a standard webcam effectively capture individual attention preferences without active user calibration?
• RQ2 How well can the integration of gaze data and computational saliency models improve the accuracy of human attention prediction in natural scenes?
• RQ3 To what extent does incremental model updating via gaze data enhance personalization over time in a passive, non-intrusive manner?
• RQ4 Can the proposed method enable practical, user-specific applications such as saliency refinement and attention-based photo retrieval?
• RQ5 How does the method compare to state-of-the-art saliency models in terms of personalization and performance on real user data?

Key findings

• The proposed method achieves accurate personal saliency maps by fusing gaze data with computational saliency, significantly reducing false positives common in standalone saliency models.
• Incremental GMM clustering enables effective user-specific model adaptation with minimal computational overhead, achieving convergence within 100 iterations.
• The system learns user attention preferences passively during normal photo viewing, requiring no explicit training or interaction steps.
• The method enables effective photo retrieval using personal attention regions as queries, with top-ranked results matching user interest locations.
• Tag suggestion applications outperform traditional face-detection-based methods by proposing tags on high-attention, non-face regions that are often missed.
• The approach demonstrates robustness across 10 subjects, with consistent performance improvements over baseline saliency models in personalized settings.