[Paper Review] Cross-convolutional-layer Pooling for Image Recognition

Lingqiao Liu, Chunhua Shen+1 more|arXiv (Cornell University)|Oct 4, 2015

Advanced Image and Video Retrieval Techniques21 references3 citations

TL;DR

This paper proposes cross-convolutional-layer pooling (CCLP), a novel method to extract image representations by leveraging activations from two consecutive convolutional layers: one for local feature extraction and the next as a pooling guidance signal. The approach improves performance on image classification and retrieval by dynamically weighting local features using feature maps as part of a generalized parts-based pooling mechanism, outperforming standard pooling and existing DCNN-based descriptors on multiple benchmarks.

ABSTRACT

Recent studies have shown that a Deep Convolutional Neural Network (DCNN) pretrained on a large image dataset can be used as a universal image descriptor, and that doing so leads to impressive performance for a variety of image classification tasks. Most of these studies adopt activations from a single DCNN layer, usually the fully-connected layer, as the image representation. In this paper, we proposed a novel way to extract image representations from two consecutive convolutional layers: one layer is utilized for local feature extraction and the other serves as guidance to pool the extracted features. By taking different viewpoints of convolutional layers, we further develop two schemes to realize this idea. The first one directly uses convolutional layers from a DCNN. The second one applies the pretrained CNN on densely sampled image regions and treats the fully-connected activations of each image region as convolutional feature activations. We then train another convolutional layer on top of that as the pooling-guidance convolutional layer. By applying our method to three popular visual classification tasks, we find our first scheme tends to perform better on the applications which need strong discrimination on subtle object patterns within small regions while the latter excels in the cases that require discrimination on category-level patterns. Overall, the proposed method achieves superior performance over existing ways of extracting image representations from a DCNN.

Motivation & Objective

• To address the limitation of using only a single layer (e.g., fully connected) for image representation in pre-trained DCNNs.
• To improve feature discrimination by exploiting spatial and hierarchical patterns across consecutive convolutional layers.
• To develop a computationally efficient method for image retrieval using cross-layer pooling with feature binarization and adaptive channel selection.
• To reduce memory and computational costs without sacrificing discriminative power through feature sign quantization.
• To demonstrate the superiority of CCLP over standard pooling and existing DCNN descriptor methods on standard benchmarks.

Proposed method

• The method extracts local features from an earlier convolutional layer and uses the feature maps of the subsequent convolutional layer as a guidance signal for pooling.
• Each filter’s feature map in the guiding layer acts as a part detector, assigning weights to local features from the prior layer based on spatial activation.
• The final image representation is formed by concatenating pooled features across all filters (channels) of the guiding layer.
• Two variants are proposed: one using original convolutional layers and another treating region-wise fully connected activations as an augmented convolutional layer.
• For image retrieval, the method applies feature binarization and selects top-k pooling channels based on average activation to reduce computational cost.
• A coarse feature sign quantization scheme is used to compress representations while preserving discriminative power.

Experimental results

Research questions

• RQ1 Can combining activations from two consecutive convolutional layers improve image representation learning compared to single-layer descriptors?
• RQ2 Does using feature maps from a later layer as a dynamic pooling guide enhance feature discrimination for fine-grained and category-level recognition?
• RQ3 How does cross-layer pooling compare to standard pooling methods like max-pooling and sum-pooling with spatial pyramid pooling?
• RQ4 Can computational cost be reduced in image retrieval without degrading performance using binarization and adaptive channel selection?
• RQ5 To what extent does feature sign quantization preserve performance in compressed representations?

Key findings

• Cross-layer pooling achieved 74.4%, 77.0%, and 84.1% accuracy on MIT67, Birds200, and PASCAL07, outperforming all alternative pooling methods tested.
• On image retrieval, the method with k=50 pooling channels and binarized features significantly outperformed the baseline method of [5], especially when features were compressed.
• Performance dropped when too many low-activation pooling channels were included, indicating noise from weak detectors.
• Feature sign quantization preserved most of the discriminative power, with only minor accuracy drops (e.g., 77.9% vs. 78.2% on MIT-67).
• The method using augmented convolutional layers (from region-wise FC activations) showed superior performance on category-level recognition tasks.
• The proposed retrieval scheme with adaptive channel selection and binarization achieved high performance at reasonable computational cost.