[Paper Review] Feature Space Augmentation for Long-Tailed Data

Peng Chu, Xiao Bian+2 more|arXiv (Cornell University)|Aug 9, 2020

Domain Adaptation and Few-Shot Learning50 references13 citations

TL;DR

This paper proposes Feature Space Augmentation (FSA), a method that generates synthetic samples for long-tailed data by fusing class-specific features from under-represented (tail) classes with class-generic features from well-represented (head) classes in deep feature space. By leveraging class activation maps to separate and recombine features at high-level network layers, FSA improves model generalization on tail classes, achieving state-of-the-art performance across CIFAR-LT, ImageNet-LT, Places-LT, and iNaturalist datasets.

ABSTRACT

Real-world data often follow a long-tailed distribution as the frequency of each class is typically different. For example, a dataset can have a large number of under-represented classes and a few classes with more than sufficient data. However, a model to represent the dataset is usually expected to have reasonably homogeneous performances across classes. Introducing class-balanced loss and advanced methods on data re-sampling and augmentation are among the best practices to alleviate the data imbalance problem. However, the other part of the problem about the under-represented classes will have to rely on additional knowledge to recover the missing information. In this work, we present a novel approach to address the long-tailed problem by augmenting the under-represented classes in the feature space with the features learned from the classes with ample samples. In particular, we decompose the features of each class into a class-generic component and a class-specific component using class activation maps. Novel samples of under-represented classes are then generated on the fly during training stages by fusing the class-specific features from the under-represented classes with the class-generic features from confusing classes. Our results on different datasets such as iNaturalist, ImageNet-LT, Places-LT and a long-tailed version of CIFAR have shown the state of the art performances.

Motivation & Objective

• Address the performance degradation of deep learning models on long-tailed datasets where tail classes suffer from insufficient training samples.
• Overcome the limitations of traditional data augmentation and class-balanced loss functions when tail classes are severely under-represented.
• Recover missing distributional information in tail classes by leveraging knowledge from head classes through feature space manipulation.
• Develop an end-to-end training pipeline that generates novel, realistic samples in the high-level feature space during training.

Proposed method

• Decompose deep features of each class into class-generic and class-specific components using class activation maps (CAM).
• For each tail class sample, augment its high-level features by mixing its class-specific features with class-generic features from confusing head classes.
• Perform feature fusion in the high-level feature space (after the last residual block) to preserve spatial structure and benefit from linear representation.
• Use a two-phase training strategy: pre-training with balanced sampling and fine-tuning with online feature-space augmented samples.
• Dynamically select head classes based on a ratio threshold (hr ≥ 0.9) to ensure sufficient generic feature representation.
• Apply augmentation after the fourth residual block in ResNet to minimize spatial noise and maximize linear feature utility.

Experimental results

Research questions

• RQ1 Can class-generic features from head classes effectively recover the missing distributional information in under-represented tail classes?
• RQ2 Does fusing class-specific features from tail classes with generic features from head classes in the deep feature space generate meaningful, generalizable samples?
• RQ3 How does the choice of head class ratio (hr) affect the performance of tail class generalization?
• RQ4 At which network depth is feature space augmentation most effective for long-tailed learning?
• RQ5 Can this method outperform existing state-of-the-art approaches like class-balanced loss and focal loss on real-world long-tailed datasets?

Key findings

• On iNaturalist 2017, the proposed method achieved a top-1 accuracy of 66.58% using ResNet-152, surpassing the previous SOTA of 64.75% using class-balanced loss.
• On iNaturalist 2018, the method achieved 69.08% top-1 accuracy with ResNet-152, outperforming the prior state-of-the-art of 66.97%.
• The optimal head class ratio (hr) was found to be around 0.95 across datasets, balancing sufficient generic feature extraction and adequate tail class coverage.
• Feature augmentation after the fourth residual block yielded the highest performance, as it minimizes spatial noise and leverages the linearity of high-level features.
• Ablation studies confirmed that feature space augmentation significantly improves tail class performance, with a clear performance gain over baseline and no-augmentation settings.