[Paper Review] TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking

Pingmei Xu, Krista A. Ehinger+4 more|arXiv (Cornell University)|Apr 25, 2015

Visual Attention and Saliency Detection29 references261 citations

TL;DR

This paper presents TurkerGaze, a crowdsourced eye tracking system using webcams on Amazon Mechanical Turk to collect high-quality saliency data at scale. By integrating gaze tracking into a gamified interface, it achieves lab-quality accuracy with significantly lower cost and effort, enabling the creation of iSUN—a large-scale saliency dataset of 20,608 natural scene images with 3 observers per image.

ABSTRACT

Traditional eye tracking requires specialized hardware, which means collecting gaze data from many observers is expensive, tedious and slow. Therefore, existing saliency prediction datasets are order-of-magnitudes smaller than typical datasets for other vision recognition tasks. The small size of these datasets limits the potential for training data intensive algorithms, and causes overfitting in benchmark evaluation. To address this deficiency, this paper introduces a webcam-based gaze tracking system that supports large-scale, crowdsourced eye tracking deployed on Amazon Mechanical Turk (AMTurk). By a combination of careful algorithm and gaming protocol design, our system obtains eye tracking data for saliency prediction comparable to data gathered in a traditional lab setting, with relatively lower cost and less effort on the part of the researchers. Using this tool, we build a saliency dataset for a large number of natural images. We will open-source our tool and provide a web server where researchers can upload their images to get eye tracking results from AMTurk.

Motivation & Objective

• To address the high cost and limited scalability of traditional lab-based eye tracking for saliency data collection.
• To develop a robust, browser-based webcam eye tracking system that functions reliably under uncontrolled conditions.
• To design a gamified interface that motivates participants to produce accurate gaze data without specialized training.
• To collect a large-scale, free-viewing saliency dataset for natural scenes using crowdsourced participants.
• To validate the quality of crowdsourced gaze data against gold-standard lab data and benchmark saliency models.

Proposed method

• Deploying a webcam-based eye tracking system within a browser-based game to collect gaze data from Mechanical Turk workers.
• Using appearance-based gaze estimation with adaptive linear regression to predict gaze from eye images, minimizing calibration needs.
• Designing two game scenarios—'Find the Target' and 'Spot the Difference'—to encourage sustained attention and accurate fixation reporting.
• Applying a real-time gaze estimation pipeline that compensates for head movement and variable lighting using synthetic training data and sparse calibration points.
• Aggregating gaze data from multiple participants per image to generate robust saliency maps and building the iSUN dataset with full object/scene annotations.
• Evaluating performance using standard metrics such as AUC, precision, recall, and fixation distribution similarity to lab data.

Experimental results

Research questions

• RQ1 Can webcam-based eye tracking on a crowdsourcing platform like Amazon Mechanical Turk produce saliency data of quality comparable to traditional lab-based eye tracking?
• RQ2 How does the gaze data collected via TurkerGaze compare quantitatively to gold-standard lab data in terms of fixation accuracy and distribution?
• RQ3 To what extent can crowdsourced gaze data from TurkerGaze be used to train and evaluate saliency prediction models?
• RQ4 How does the demographic and behavioral diversity of Mechanical Turk participants affect the reliability and generalizability of the collected gaze data?
• RQ5 Can the proposed system be generalized to other visual stimuli and tasks beyond free-viewing natural scenes?

Key findings

• The TurkerGaze system achieved gaze prediction accuracy comparable to traditional lab-based eye tracking, with a mean absolute error of 1.05° in fixation estimation.
• Fixation distributions collected via TurkerGaze were more central than those from lab data, likely due to differences in viewing behavior or image presentation.
• Leave-one-out AUC for saliency prediction using AMTurk data was lower than for Judd lab data, indicating higher positional noise in crowdsourced data.
• Despite noise, saliency maps derived from TurkerGaze data achieved AUC scores comparable to state-of-the-art computational models, demonstrating their utility for training and evaluation.
• The iSUN dataset, built using TurkerGaze, contains 20,608 fully-annotated natural scene images with an average of 3 observers per image, making it the largest free-viewing saliency dataset to date.
• The system successfully reduced the need for calibration and enabled real-time gaze tracking on standard consumer hardware, enabling scalable data collection.