[Paper Review] Between-class Learning for Image Classification

Yuji Tokozume, Yoshitaka Ushiku+1 more|arXiv (Cornell University)|Nov 28, 2017

Domain Adaptation and Few-Shot Learning25 references6 citations

TL;DR

This paper proposes Between-Class (BC) learning for image classification, where models are trained to predict the mixing ratio of two images from different classes. By treating images as waveforms and applying simple or energy-aware mixing, BC learning improves generalization, reducing ImageNet-1K top-1 error to 19.4% and CIFAR-10 error to 2.26%.

ABSTRACT

In this paper, we propose a novel learning method for image classification called Between-Class learning (BC learning). We generate between-class images by mixing two images belonging to different classes with a random ratio. We then input the mixed image to the model and train the model to output the mixing ratio. BC learning has the ability to impose constraints on the shape of the feature distributions, and thus the generalization ability is improved. BC learning is originally a method developed for sounds, which can be digitally mixed. Mixing two image data does not appear to make sense; however, we argue that because convolutional neural networks have an aspect of treating input data as waveforms, what works on sounds must also work on images. First, we propose a simple mixing method using internal divisions, which surprisingly proves to significantly improve performance. Second, we propose a mixing method that treats the images as waveforms, which leads to a further improvement in performance. As a result, we achieved 19.4% and 2.26% top-1 errors on ImageNet-1K and CIFAR-10, respectively.

Motivation & Objective

• To extend Between-Class learning, originally developed for sound recognition, to image classification.
• To investigate whether mixing images from different classes can improve model generalization despite visual incoherence.
• To explore whether CNNs implicitly treat images as waveforms, enabling effective mixing-based training.
• To develop a simple yet effective mixing strategy that enhances feature distribution constraints.
• To empirically validate BC learning across diverse architectures and datasets, demonstrating consistent performance gains.

Proposed method

• Generate mixed images by combining two images from different classes using a random mixing ratio.
• Train the model to regress the mixing ratio using a regression head, introducing a new ratio prediction head.
• Apply a simple mixing method using internal divisions (e.g., spatial cropping and blending) to create mixed inputs.
• Propose BC+ by treating images as zero-mean waveforms: subtract per-image mean and normalize by image energy before mixing.
• Use the same mixing ratio for both input and supervision, enabling end-to-end training with a single loss.
• Introduce a multi-class mixing strategy where the model learns to predict the ratio even when multiple classes are involved in the mixture.

Experimental results

Research questions

• RQ1 Can Between-Class learning, effective in sound recognition, be successfully applied to image classification despite visual incoherence?
• RQ2 Does treating images as waveforms enable meaningful data mixing that improves model generalization?
• RQ3 How does the choice of mixing location within the network (e.g., input vs. feature layers) affect performance?
• RQ4 What is the optimal number of classes to mix (e.g., two vs. three) for maximizing performance?
• RQ5 Does BC learning impose structural constraints on feature distributions that lead to better generalization?

Key findings

• BC learning reduced the top-1 error rate on ImageNet-1K from 20.4% to 19.4% using ResNeXt-101 (64×4d).
• The improved BC+ method reduced the error rate on CIFAR-10 from 2.86% to 2.26% using Shake-Shake Regularization.
• The ablation study showed that mixing two images from different classes (N=2) consistently outperformed random or same-class mixing.
• Mixing at the input layer yielded the best performance, while mixing at deeper layers (e.g., pool2) degraded results.
• Feature visualization via 3D PCA confirmed that BC learning produces more spherical, tightly clustered feature distributions with reduced within-class variance.
• The method achieved consistent improvements across multiple architectures, including a simple 11-layer CNN and state-of-the-art models.