[Paper Review] Efficient Egocentric Visual Perception Combining Eye-tracking, a Software Retina and Deep Learning

Nina Hristozova, Piotr Ozimek+1 more|ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam)|Sep 5, 2018

CCD and CMOS Imaging Sensors5 references4 citations

TL;DR

This paper proposes an efficient egocentric visual perception system that uses human eye-tracking data to guide a software retina model, reducing input data by a factor of 16.7 while achieving 98.2% test accuracy on a 9-class object recognition task via a custom DCNN, demonstrating high performance with significantly reduced computational load through bio-inspired foveated vision and deep learning integration.

ABSTRACT

We present ongoing work to harness biological approaches to achieving highly efficient egocentric perception by combining the space-variant imaging architecture of the mammalian retina with Deep Learning methods. By pre-processing images collected by means of eye-tracking glasses to control the fixation locations of a software retina model, we demonstrate that we can reduce the input to a DCNN by a factor of 3, reduce the required number of training epochs and obtain over 98% classification rates when training and validating the system on a database of over 26,000 images of 9 object classes.

Motivation & Objective

• To develop an efficient egocentric perception system that leverages human eye-tracking data to guide visual sampling.
• To integrate a high-resolution software retina model with deep learning for reduced input data and improved computational efficiency.
• To demonstrate state-of-the-art classification performance using fixation-guided, space-variant image sampling.
• To enable unconstrained data collection by replacing rigid head stabilization with predictive fixation modeling.
• To explore the use of conformal mapping and cortical image transformation for scale- and rotation-invariant feature extraction.

Proposed method

• Eye-tracking glasses (Tobii Pro 2) captured images while human observers fixated on salient object regions.
• Fixation locations were clustered using K-means (K = 1% of fixations per class) to identify representative foveal regions.
• A 50K-node software retina model applied complex-log conformal mapping to transform full images into space-variant cortical images of size 399×752px.
• A custom DCNN with seven 3×3 convolutional layers, max pooling, and three 132-node fully connected layers was trained on cortical images.
• The system compared performance using cortical images versus full-resolution fixation crops to benchmark data efficiency and accuracy.
• A scattered datapoint gridding algorithm was applied to cortical images to further reduce input size while maintaining accuracy.

Figure 1: Left, Tobii Pro2 Eye Tracker Glasses; Right, Fixation clustering following homography alignment

Experimental results

Research questions

• RQ1 Can human eye-tracking data be effectively used to guide a software retina for efficient egocentric object recognition?
• RQ2 To what extent does the software retina’s space-variant sampling reduce input data while preserving classification performance?
• RQ3 How does the performance of a DCNN trained on cortical images compare to one trained on full-resolution fixation crops?
• RQ4 Can a custom network layer directly connected to retina outputs improve efficiency beyond generating intermediate cortical images?
• RQ5 Can an auxiliary network predict diagnostic fixation points and enable unconstrained data collection in egocentric settings?

Key findings

• The system achieved 98.2% test accuracy on a 9-class object recognition task using cortical images derived from eye-tracking data.
• A 16.7-fold reduction in visual data input was achieved through the software retina’s space-variant sampling, with minimal accuracy loss.
• The cortical image-based DCNN required only 6 seconds to classify the test set, compared to 12 seconds for the full-resolution fixation crop model.
• The full-resolution fixation crop model achieved 99.5% test accuracy but required larger batch sizes and longer training time.
• Subsampling cortical images with a scattered datapoint gridding algorithm achieved a 10.8× input data reduction without performance loss.
• An auxiliary network is being developed to predict diagnostic fixations and enable object segmentation from arbitrary, partially occluded views.