[Paper Review] CINIC-10 is not ImageNet or CIFAR-10

Luke Nicholas Darlow, Elliot J. Crowley+2 more|arXiv (Cornell University)|Oct 2, 2018

Image Retrieval and Classification TechniquesComputer Science168 citations

TL;DR

CINIC-10 augments CIFAR-10 with downsampled ImageNet images to form a larger, equally split dataset suitable as a drop-in CIFAR-10 replacement while introducing distribution shift between CIFAR and ImageNet samples.

ABSTRACT

In this brief technical report we introduce the CINIC-10 dataset as a plug-in extended alternative for CIFAR-10. It was compiled by combining CIFAR-10 with images selected and downsampled from the ImageNet database. We present the approach to compiling the dataset, illustrate the example images for different classes, give pixel distributions for each part of the repository, and give some standard benchmarks for well known models. Details for download, usage, and compilation can be found in the associated github repository.

Motivation & Objective

• Motivate benchmarking beyond CIFAR-10 and ImageNet by providing a larger, intermediate-difficulty dataset.
• Describe a method to construct CINIC-10 by augmenting CIFAR-10 with ImageNet-derived images.
• Characterize distribution differences between CIFAR and ImageNet samples.
• Provide standard benchmarks to establish baseline performance on CINIC-10.

Proposed method

• Reformat CIFAR-10 images into PNG format and create equal train/validation/test splits with 20,000 CIFAR-10 images per set.
• Identify and use synsets from ImageNet Fall 2011 corresponding to CIFAR-10 classes.
• Convert ImageNet JPEGs to 32×32 color images and randomly sample 21,000 images per class from synset groups to augment CIFAR-10.
• Distribute the 21000 ImageNet samples across train/validation/test sets with filenames encoding origin and synset information.
• Provide usage guidelines and a PyTorch data-loading example with normalization means and stds.

Experimental results

Research questions

• RQ1 How does CINIC-10 compare to CIFAR-10 and ImageNet in terms of dataset size and class distribution?
• RQ2 What is the effect of distribution shift when training on CIFAR-10 images and testing on CINIC-10 (ImageNet-contributed samples)?
• RQ3 What baseline model performance can be expected on CINIC-10 using standard architectures?
• RQ4 Can CINIC-10 serve as a practical drop-in replacement for CIFAR-10 with comparable preprocessing?

Key findings

• CINIC-10 contains 270,000 images, 60,000 CIFAR-10 images plus 210,000 ImageNet-derived images, with equally sized train/validation/test splits.
• Each class has 9,000 images per split, totaling 27,000 per class across the three splits.
• CINIC-10 images exhibit distribution shifts between CIFAR and ImageNet samples, with similar but not identical color distributions.
• Benchmark results (test error) for standard models on CINIC-10 include: VGG-16 12.23% ± 0.16, ResNet-18 9.73% ± 0.05, GoogLeNet 8.83% ± 0.12, ResNeXt29_2x64d 8.55% ± 0.15, DenseNet-121 8.74% ± 0.16, MobileNet 18.00% ± 0.16.