[Paper Review] Pruning Convolutional Neural Networks for Image Instance Retrieval

Gaurav Manek, Jie Lin+5 more|arXiv (Cornell University)|Jul 18, 2017

Advanced Image and Video Retrieval Techniques20 references3 citations

TL;DR

This paper proposes a heuristic-based pruning framework combined with triplet loss fine-tuning to compress convolutional neural networks for image instance retrieval. It achieves up to 5× model compression with minimal performance drop by pruning convolutional edges using data-independent and data-dependent criteria, followed by end-to-end fine-tuning on retrieval-specific loss.

ABSTRACT

In this work, we focus on the problem of image instance retrieval with deep descriptors extracted from pruned Convolutional Neural Networks (CNN). The objective is to heavily prune convolutional edges while maintaining retrieval performance. To this end, we introduce both data-independent and data-dependent heuristics to prune convolutional edges, and evaluate their performance across various compression rates with different deep descriptors over several benchmark datasets. Further, we present an end-to-end framework to fine-tune the pruned network, with a triplet loss function specially designed for the retrieval task. We show that the combination of heuristic pruning and fine-tuning offers 5x compression rate without considerable loss in retrieval performance.

Motivation & Objective

• Address the challenge of reducing model size in CNNs for image instance retrieval without sacrificing retrieval accuracy.
• Investigate the effectiveness of heuristic pruning strategies on convolutional layers, which are more complex to compress than fully-connected layers.
• Develop and evaluate an end-to-end framework that combines pruning with retrieval-specific fine-tuning to restore performance after compression.
• Assess the impact of different pruning heuristics and compression rates across multiple benchmark datasets and deep descriptor types.
• Determine the optimal trade-off between model compression and retrieval performance for real-world deployment in mobile and embedded systems.

Proposed method

• Apply data-independent heuristics—such as weight magnitude |wᵢⱼ|, gradient-weighted salience (dℒ/dwᵢⱼ × wᵢⱼ), mean activation × weight, and variance of activation × weight squared—to assign importance scores to convolutional edges.
• Use a data-dependent heuristic based on the Taylor expansion of loss change to identify less critical edges for removal.
• Implement an end-to-end fine-tuning pipeline using a triplet loss function specifically designed for image instance retrieval to recover performance after pruning.
• Apply pruning to convolutional layers only, preserving feature hierarchy and focusing on redundancy in higher layers.
• Use SQP (Spherical Quantized Pooling) to generate deep descriptors from pruned features for evaluation.
• Evaluate performance across multiple datasets (Oxford5k, Paris6k, Holidays, UKbench) at varying compression rates (10%, 20%, 50% of original size).

Experimental results

Research questions

• RQ1 Can heuristic pruning of convolutional layers maintain high retrieval performance when applied to image instance retrieval tasks?
• RQ2 How do different pruning heuristics (e.g., weight magnitude, gradient-based, activation-based) compare in terms of performance retention at various compression rates?
• RQ3 To what extent can fine-tuning with a triplet loss function restore performance after aggressive pruning of convolutional layers?
• RQ4 What is the practical limit of compression (i.e., minimum model size) before performance degrades significantly?
• RQ5 Does the pattern of layer-wise pruning vary across network depth, and does it correlate with redundancy distribution?

Key findings

• Heuristic pruning using weight magnitude (|wᵢⱼ|) achieves up to 2× compression with minimal performance loss on all benchmark datasets.
• Combining heuristic pruning with end-to-end fine-tuning using triplet loss enables up to 5× compression while maintaining high retrieval accuracy, with mean average precision (mAP) reaching ~80% on Oxford5k and Paris6k after 50% pruning.
• Fine-tuning significantly improves performance on Oxford5k and Paris6k, but slightly reduces recall@4 on the UKbench dataset, likely due to domain shift from training data.
• Networks pruned to 10% of original size show a performance discontinuity, indicating that further pruning beyond 20% may exceed the model's redundancy threshold.
• Layer-wise pruning shows that lower layers retain ~95% of edges even at 10% network size, while higher layers lose up to 92% of edges, confirming higher redundancy in deeper layers.
• The 20–10% range represents a performance threshold where further compression leads to disproportionate performance degradation, suggesting a practical lower bound for effective pruning.