[Paper Review] Gaze Embeddings for Zero-Shot Image Classification

Nour Karessli, Zeynep Akata+2 more|arXiv (Cornell University)|Nov 28, 2016

Domain Adaptation and Few-Shot Learning36 references4 citations

TL;DR

This paper proposes using human gaze data as auxiliary information for zero-shot image classification, introducing three novel gaze embeddings—Gaze Histograms (GH), Gaze Features with Grid (GFG), and Gaze Features with Sequence (GFS)—to capture class-discriminative visual attention. The method outperforms expert-annotated attributes and baseline methods on fine-grained datasets, demonstrating that non-expert gaze is a competitive, low-cost alternative for zero-shot learning.

ABSTRACT

Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts. We instead propose a method that relies on human gaze as auxiliary information, exploiting that even non-expert users have a natural ability to judge class membership. We present a data collection paradigm that involves a discrimination task to increase the information content obtained from gaze data. Our method extracts discriminative descriptors from the data and learns a compatibility function between image and gaze using three novel gaze embeddings: Gaze Histograms (GH), Gaze Features with Grid (GFG) and Gaze Features with Sequence (GFS). We introduce two new gaze-annotated datasets for fine-grained image classification and show that human gaze data is indeed class discriminative, provides a competitive alternative to expert-annotated attributes, and outperforms other baselines for zero-shot image classification.

Motivation & Objective

• To address the high cost and expertise requirement of collecting expert-annotated attributes for zero-shot image classification.
• To explore whether human gaze—collected from non-experts—can serve as a viable, low-cost alternative to attributes for zero-shot learning.
• To develop a novel data collection paradigm that enhances gaze information content by using a discrimination task between two classes.
• To design and evaluate three new gaze embedding methods that extract discriminative features from gaze point sequences.
• To demonstrate generalization of gaze-based zero-shot learning across different fine-grained datasets beyond the initial training domain.

Proposed method

• Proposes a gaze data collection paradigm where observers view two classes side-by-side and perform a binary classification task on a third image, encouraging focus on discriminative object parts.
• Introduces three gaze embedding methods: Gaze Histograms (GH) for spatial distribution of gaze points, Gaze Features with Grid (GFG) for spatial and temporal features in a grid, and Gaze Features with Sequence (GFS) for sequential gaze patterns.
• Uses a structured joint embedding framework to learn compatibility between deep image features and gaze embeddings for zero-shot classification.
• Applies gaze embeddings in a zero-shot learning setup by learning a mapping between image features and class embeddings using gaze as auxiliary signal.
• Employs eye-tracking data to extract gaze points and constructs embeddings based on spatial distribution, fixation duration, and sequence order.
• Validates performance using standard zero-shot learning protocols on CUB and PET datasets with multiple zero-shot splits and repeated experiments.

Experimental results

Research questions

• RQ1 Can human gaze data from non-experts serve as effective auxiliary information for zero-shot image classification?
• RQ2 How do gaze embeddings compare to expert-annotated attributes and other baselines in zero-shot classification performance?
• RQ3 Does a discrimination-based data collection paradigm improve the discriminative quality of gaze data compared to passive viewing?
• RQ4 Can gaze embeddings generalize across different fine-grained datasets such as CUB and Oxford Pets?
• RQ5 What are the limitations of gaze-based embeddings in distinguishing visually similar classes, such as cat breeds with similar shapes but different colors?

Key findings

• Human gaze data from non-experts is highly class-discriminative and achieves a zero-shot accuracy of 46.6% on the PET dataset, significantly outperforming random baseline (16%) and bag-of-words (21.0%) baselines.
• On the CUB-VW dataset, gaze embeddings outperform both expert-annotated attributes and mouse-click-based 'bubble' data, demonstrating their superiority as auxiliary signals.
• Gaze embeddings achieve competitive performance on fine-grained bird classification, correctly ranking 'Black-capped Vireo' images in the top three positions, while attributes and bag-of-words methods make errors due to semantic similarity in text.
• The qualitative analysis shows gaze embeddings rarely confuse cats and dogs, whereas bag-of-words representations frequently do, highlighting gaze's robustness to semantic ambiguity.
• Failure cases occur when distinguishing visually similar classes (e.g., Abyssinian vs. Russian Blue cats), where gaze fails to encode color differences, indicating a limitation in capturing non-spatial visual attributes.
• The results suggest that gaze embeddings are effective for zero-shot learning and generalize across domains, though performance may improve with domain-specific data collection or fine-tuning of gaze embedding parameters.