[Paper Review] Faster gaze prediction with dense networks and Fisher pruning

Lucas Theis, Iryna Korshunova+2 more|arXiv (Cornell University)|Jan 17, 2018

Visual Attention and Saliency DetectionComputer Science4 references138 citations

TL;DR

The paper introduces Fisher pruning to greedily remove redundant feature maps and parameters from gaze-prediction models, achieving about 10x CPU speedups with comparable saliency performance, by combining pruning with knowledge distillation.

ABSTRACT

Predicting human fixations from images has recently seen large improvements by leveraging deep representations which were pretrained for object recognition. However, as we show in this paper, these networks are highly overparameterized for the task of fixation prediction. We first present a simple yet principled greedy pruning method which we call Fisher pruning. Through a combination of knowledge distillation and Fisher pruning, we obtain much more runtime-efficient architectures for saliency prediction, achieving a 10x speedup for the same AUC performance as a state of the art network on the CAT2000 dataset. Speeding up single-image gaze prediction is important for many real-world applications, but it is also a crucial step in the development of video saliency models, where the amount of data to be processed is substantially larger.

Motivation & Objective

• Motivate the need for faster gaze prediction models due to overparameterization in transfer-learned saliency networks.
• Develop a principled pruning method (Fisher pruning) to remove redundant feature maps/parameters.
• Combine pruning with knowledge distillation to retain performance while reducing runtime.
• Provide a framework to balance performance and computational cost via a tunable trade-off parameter.
• Demonstrate end-to-end training and pruning can yield competitive or superior generalization on saliency benchmarks.

Proposed method

• Build on DeepGaze II with VGG or DenseNet backbones and a readout network to produce saliency maps.
• Derive Fisher pruning: estimate loss increase from removing a parameter using a second-order approximation and Fisher information (Equation 7).
• Extend pruning to prune whole feature maps by aggregating per-map gradients (Δk).
• Introduce a cost-penalized objective that trades off cross-entropy loss and computational cost via a beta parameter (Equation 12).
• Propose automatic tuning of beta to prune features with negative combined pruning signal (Equation 14–15).
• Train with knowledge distillation from an ensemble of DeepGaze II models to improve feature representations and finetune while pruning.

Experimental results

Research questions

• RQ1 Can Fisher pruning provide a principled, gradient-based criterion to prune neural network components for saliency without retraining from scratch?
• RQ2 How much can saliency models be pruned (in feature maps and parameters) before degrading performance on standard benchmarks?
• RQ3 Does incorporating a computation-cost penalty (beta) yield more efficient architectures with maintained saliency performance?
• RQ4 Can knowledge distillation help recover performance when fine-tuning pruned, over-parameterized saliency models?
• RQ5 What are the comparative runtime and accuracy trade-offs for FastGaze and DenseGaze under pruning?

Key findings

• Fisher pruning yields competitive saliency performance while significantly reducing computation, achieving up to around 10x CPU speedups for similar AUC on CAT2000.
• Regularizing computational cost during pruning improves performance over unregularized methods and updating feature-map costs during pruning is beneficial.
• Pruned FastGaze and DenseGaze models generalize to CAT2000 comparably to or better than DeepGaze II across metrics like log-likelihood, NSS, and SIM, while maintaining or improving AUC.
• DenseGaze achieves higher AUC with pruning; FastGaze offers faster runtimes due to simpler architecture.
• Training with knowledge distillation from DeepGaze II ensembles assists in finetuning pruned models and maintaining performance.
• Heavily pruned models can reach speedups up to 39x versus DeepGaze II with saliency maps still capturing key structures like faces and text.