[Paper Review] Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition

Heliang Zheng, Jianlong Fu+2 more|arXiv (Cornell University)|Mar 14, 2019

Advanced Neural Network Applications37 references38 citations

TL;DR

The paper introduces Trilinear Attention Sampling Network (TASN) for fine-grained image recognition, learning fine-grained details from hundreds of part proposals via a teacher–student distillation approach, and achieves state-of-the-art results on CUB-200-2011, Stanford Cars, and iNaturalist 2017.

ABSTRACT

Learning subtle yet discriminative features (e.g., beak and eyes for a bird) plays a significant role in fine-grained image recognition. Existing attention-based approaches localize and amplify significant parts to learn fine-grained details, which often suffer from a limited number of parts and heavy computational cost. In this paper, we propose to learn such fine-grained features from hundreds of part proposals by Trilinear Attention Sampling Network (TASN) in an efficient teacher-student manner. Specifically, TASN consists of 1) a trilinear attention module, which generates attention maps by modeling the inter-channel relationships, 2) an attention-based sampler which highlights attended parts with high resolution, and 3) a feature distiller, which distills part features into a global one by weight sharing and feature preserving strategies. Extensive experiments verify that TASN yields the best performance under the same settings with the most competitive approaches, in iNaturalist-2017, CUB-Bird, and Stanford-Cars datasets.

Motivation & Objective

• Motivate learning discriminative fine-grained details (e.g., beak, eyes) beyond limited pre-defined parts.
• Develop a scalable method to extract hundreds of part proposals without part annotations.
• Propose an efficient teacher–student framework to distill numerous part features into a single backbone.
• Demonstrate effectiveness across major FGVC datasets with competitive efficiency.

Proposed method

• Introduce a trilinear attention module to generate inter-channel attention maps from convolutional features.
• Develop an attention-based sampler that creates structure-preserved and detail-preserved images to capture global and part-specific details.
• Propose a feature distiller where a part-net (teacher) guides a master-net (student) via soft-target cross-entropy to fuse hundreds of part details into a single network.
• Adopt knowledge distillation to transfer fine-grained details from part-net to master-net, enabling efficient single-stream inference.

Experimental results

Research questions

• RQ1 Can learning from hundreds of part proposals improve fine-grained recognition without explicit part annotations?
• RQ2 How effective is trilinear attention in generating robust, high-resolution part maps for sampling fine-grained details?
• RQ3 Does the proposed teacher–student distillation scheme efficiently integrate many part features into a single backbone without performance loss?
• RQ4 What are the gains of TASN across standard FGVC datasets compared to state-of-the-art baselines?

Key findings

• TASN achieves strong accuracy gains over baselines on CUB-200-2011, Stanford Cars, and iNaturalist 2017 under comparable backbones and single-stream inference.
• On CUB-200-2011 with ResNet-50 at 224 input, TASN reaches 87.0% accuracy (vs 81.6% for the baseline ResNet-50).
• With ResNet-50 at 448 input, TASN attains 87.0% (vs 85.6% for higher-resolution baseline), and outperforms SSN- and DT-RAM-based approaches.
• On Stanford Cars, TASN with 1x VGG-19 achieves 92.4% (vs 88.6% baseline), and ensembles reach 93.2%.
• On iNaturalist 2017, TASN achieves 68.2% accuracy, surpassing ResNet (59.6%) and SSN (65.2%) across most superclasses.
• Overall, a single-stream TASN can outperform multi-stream part-based models and remains competitive with recent second-order feature methods when combined.