[Paper Review] Low-rank Bilinear Pooling for Fine-Grained Classification

Shu Kong, Charless C. Fowlkes|arXiv (Cornell University)|Nov 16, 2016

Domain Adaptation and Few-Shot Learning31 references4 citations

TL;DR

This paper proposes Low-Rank Bilinear Pooling (LRBP), a compact and efficient method for fine-grained image classification that replaces explicit high-dimensional bilinear feature computation with a low-rank bilinear classifier using the Frobenius norm as a scoring function. The model achieves state-of-the-art performance on benchmark datasets with only category labels, while being one to two orders of magnitude smaller than prior methods.

ABSTRACT

Pooling second-order local feature statistics to form a high-dimensional bilinear feature has been shown to achieve state-of-the-art performance on a variety of fine-grained classification tasks. To address the computational demands of high feature dimensionality, we propose to represent the covariance features as a matrix and apply a low-rank bilinear classifier. The resulting classifier can be evaluated without explicitly computing the bilinear feature map which allows for a large reduction in the compute time as well as decreasing the effective number of parameters to be learned. To further compress the model, we propose classifier co-decomposition that factorizes the collection of bilinear classifiers into a common factor and compact per-class terms. The co-decomposition idea can be deployed through two convolutional layers and trained in an end-to-end architecture. We suggest a simple yet effective initialization that avoids explicitly first training and factorizing the larger bilinear classifiers. Through extensive experiments, we show that our model achieves state-of-the-art performance on several public datasets for fine-grained classification trained with only category labels. Importantly, our final model is an order of magnitude smaller than the recently proposed compact bilinear model, and three orders smaller than the standard bilinear CNN model.

Motivation & Objective

• Address the high computational and memory cost of standard bilinear pooling in fine-grained classification, which produces very high-dimensional features.
• Reduce model size and inference time without sacrificing accuracy by replacing explicit feature map computation with a low-rank bilinear classifier.
• Enable efficient training and deployment on memory-constrained devices by introducing a co-decomposition technique that factorizes per-class classifiers into shared and class-specific components.
• Achieve state-of-the-art performance using only category-level supervision, eliminating the need for costly part or keypoint annotations.

Proposed method

• Represent the bilinear feature map as a covariance matrix and apply a low-rank constraint to the bilinear classifier, reducing the number of learnable parameters.
• Use the Frobenius norm of projected features as the classification score, enabling fast inference without explicitly computing the full bilinear feature vector.
• Introduce a co-decomposition method that factorizes the joint classifier into a shared low-rank factor and compact per-class terms, further compressing the model.
• Implement the co-decomposition via two convolutional layers in an end-to-end trainable architecture, initialized using PCA to avoid separate pre-training.
• Leverage symmetric bilinear forms to maintain structural information while enabling efficient optimization and parameter sharing.

Experimental results

Research questions

• RQ1 Can a low-rank bilinear classifier achieve comparable or better performance than full-bilinear pooling while drastically reducing model size and inference cost?
• RQ2 Does using the Frobenius norm as a scoring function enable efficient inference without explicit computation of high-dimensional bilinear features?
• RQ3 Can co-decomposition of classifier parameters further compress the model while preserving performance in fine-grained classification?
• RQ4 Can a model trained only on category labels outperform methods that rely on part or keypoint annotations?

Key findings

• The proposed LRBP model achieves state-of-the-art performance on multiple public fine-grained classification benchmarks, including CUB-200-2011, without using any part or keypoint annotations.
• The final model is one to two orders of magnitude smaller than the compact bilinear model [8] and three orders smaller than the standard bilinear CNN [20].
• The model enables fast inference by avoiding explicit computation of bilinear features, especially beneficial when the number of channels exceeds spatial resolution.
• Qualitative visualizations show the model naturally focuses on discriminative parts of birds, such as red wings or yellow heads, even without explicit spatial attention or part supervision.
• The proposed initialization via PCA followed by end-to-end fine-tuning avoids the need for pre-training and achieves strong performance with minimal hyperparameter tuning.