[Paper Review] Semantic Bilinear Pooling for Fine-Grained Recognition

Xinjie Li, Chun Yang+3 more|arXiv (Cornell University)|Apr 3, 2019

Advanced Neural Network Applications30 references4 citations

TL;DR

This paper proposes Semantic Bilinear Pooling (SBP), a two-branch deep learning framework that integrates hierarchical label structures with bilinear pooling for fine-grained recognition. By leveraging a coarse-to-fine training strategy with a generalized cross-entropy loss, SBP improves feature discrimination and achieves state-of-the-art accuracy on four benchmark datasets without inference overhead.

ABSTRACT

Naturally, fine-grained recognition, e.g., vehicle identification or bird classification, has specific hierarchical labels, where fine categories are always harder to be classified than coarse categories. However, most of the recent deep learning based methods neglect the semantic structure of fine-grained objects and do not take advantage of the traditional fine-grained recognition techniques (e.g. coarse-to-fine classification). In this paper, we propose a novel framework with a two-branch network (coarse branch and fine branch), i.e., semantic bilinear pooling, for fine-grained recognition with a hierarchical label tree. This framework can adaptively learn the semantic information from the hierarchical levels. Specifically, we design a generalized cross-entropy loss for the training of the proposed framework to fully exploit the semantic priors via considering the relevance between adjacent levels and enlarge the distance between samples of different coarse classes. Furthermore, our method leverages only the fine branch when testing so that it adds no overhead to the testing time. Experimental results show that our proposed method achieves state-of-the-art performance on four public datasets.

Motivation & Objective

• Address the challenge of fine-grained recognition, where subtle inter-class differences and intra-class variations hinder accurate classification.
• Incorporate hierarchical semantic label structures (e.g., make and model) into deep learning models to improve feature learning.
• Develop a two-branch network (coarse and fine branches) that leverages semantic priors during training while maintaining efficient inference using only the fine branch.
• Design a generalized cross-entropy loss to enhance discrimination between coarse classes and exploit label hierarchy during training.
• Achieve state-of-the-art performance on multiple fine-grained recognition benchmarks without increasing model complexity or inference time.

Proposed method

• Propose a two-branch CNN architecture: a coarse branch for hierarchical label supervision and a fine branch for final classification.
• Apply bilinear pooling using feature maps from two separate streams (C-Net and F-Net) to capture high-order feature interactions.
• Introduce a generalized cross-entropy loss that models label relevance across adjacent hierarchical levels and increases inter-coarse-class margins.
• Use a signed square root and L2 normalization after bilinear pooling to stabilize training and improve generalization.
• Retain only the fine branch during inference, ensuring no additional computational cost compared to standard models.
• Integrate hierarchical label trees (e.g., make → model) into the training process to guide feature learning with semantic priors.

Experimental results

Research questions

• RQ1 Can hierarchical label structures be effectively leveraged in deep learning models to improve fine-grained recognition?
• RQ2 How does incorporating coarse-to-fine supervision through a two-branch network affect feature discrimination and model accuracy?
• RQ3 To what extent does a generalized cross-entropy loss that models label hierarchy improve performance over standard cross-entropy?
• RQ4 Can bilinear pooling be effectively combined with semantic priors without increasing inference time or model complexity?
• RQ5 Does the proposed method outperform existing state-of-the-art approaches on standard fine-grained recognition benchmarks?

Key findings

• The proposed method achieves 97.8% accuracy on CompCars, 94.3% on Stanford Cars, 88.9% on CUB-200-2011, and 91.7% on FG-WILDBERG, outperforming all prior methods on these benchmarks.
• Using iSQRT-COV for dimensionality reduction in bilinear pooling yields the best performance, with 97.0% accuracy on CompCars and 91.1% on Aircrafts.
• The generalized cross-entropy loss contributes more significantly to performance gains than the two-branch structure alone, especially in reducing inter-coarse-class confusion.
• The model maintains inference efficiency by using only the fine branch at test time, adding no computational overhead compared to baseline models.
• The method outperforms HSE [8], which uses a four-level hierarchy and additional convolutional layers, by 0.7% on Stanford Cars despite using only a two-level hierarchy.
• Visualizations show that the attention maps are more robust and focused on discriminative parts than those from CBP [19], especially under pose and background variations.