[Paper Review] Context Augmentation for Convolutional Neural Networks

Ayşegül Dündar, Ignacio Garcia‐Dorado|arXiv (Cornell University)|Nov 22, 2017

Advanced Neural Network Applications24 references3 citations

TL;DR

This paper proposes context-aware data augmentation for convolutional neural networks (ConvNets) by separating foreground objects from backgrounds and applying targeted augmentations. By training on diverse background contexts and using segmented augmentations (e.g., flipping, translating only the foreground), the method improves image classification accuracy by 25.5%—from 47.4% to 59.5%—when using only 500 labeled images, demonstrating significant gains in low-data regimes.

ABSTRACT

Recent enhancements of deep convolutional neural networks (ConvNets) empowered by enormous amounts of labeled data have closed the gap with human performance for many object recognition tasks. These impressive results have generated interest in understanding and visualization of ConvNets. In this work, we study the effect of background in the task of image classification. Our results show that changing the backgrounds of the training datasets can have drastic effects on testing accuracies. Furthermore, we enhance existing augmentation techniques with the foreground segmented objects. The findings of this work are important in increasing the accuracies when only a small dataset is available, in creating datasets, and creating synthetic images.

Motivation & Objective

• To investigate how background context influences ConvNet performance in image classification.
• To address the challenge of low accuracy in small labeled datasets by enhancing data augmentation techniques.
• To develop new augmentation strategies that leverage foreground-background segmentation for improved generalization.
• To enable synthetic dataset creation and better data curation with minimal labeled data.

Proposed method

• The authors segment images from the STL-10 dataset into foreground objects and backgrounds using manual parsing.
• They create augmented training data by combining foreground objects with backgrounds from the same category, different categories, or replaced with mean background values.
• Segmented data augmentation is applied by performing transformations (flip, translation, rotation) only on the foreground object while preserving the background.
• Standard augmentation techniques (e.g., random flip, translation, rotation) are applied to the full image as baseline comparisons.
• The method combines standard and segmented augmentations to maximize diversity and performance.
• Experiments compare accuracy across different background configurations and augmentation strategies on 500 and 5000-image subsets of STL-10.

Experimental results

Research questions

• RQ1 How does the background context affect the accuracy of ConvNets when training data is limited?
• RQ2 Can foreground-background separation improve generalization in small-data image classification?
• RQ3 Which augmentation techniques—standard or segmented—yield higher accuracy when combined with limited labeled data?
• RQ4 Does using backgrounds from the same or different categories as the foreground object impact model performance?

Key findings

• Training with only 500 images achieved a baseline accuracy of 47.4%, which improved to 59.5% using combined standard and segmented augmentations, a 25.5% relative improvement.
• Using backgrounds from the same category as the foreground object increased accuracy to 54.4% compared to the baseline.
• Segmented horizontal flipping improved accuracy to 55.4%, while segmented translation further increased it to 58.9%.
• The final model using all effective augmentations (excluding rotation, which did not improve performance) achieved 59.5% accuracy with 500 images.
• Standard augmentations outperformed segmented versions individually, but combining both boosted performance further.
• Background context plays a critical role in classification; in some cases, the background alone provided sufficient cues for correct prediction.