[Paper Review] Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Xiao Han, Kashif Rasul+1 more|arXiv (Cornell University)|Aug 25, 2017

Advanced Neural Network Applications3 references6,056 citations

TL;DR

Fashion-MNIST introduces 70,000 28x28 grayscale fashion images across 10 classes to serve as a drop-in MNIST replacement for benchmarking ML algorithms, with a provided baseline benchmark across many classifiers.

ABSTRACT

We present Fashion-MNIST, a new dataset comprising of 28x28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. Fashion-MNIST is intended to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits. The dataset is freely available at https://github.com/zalandoresearch/fashion-mnist

Motivation & Objective

• Provide a drop-in, accessible benchmark dataset compatible with MNIST that is more challenging for modern ML methods.
• Describe the data pipeline from Zalando fashion thumbnails to 28x28 grayscale images.
• Establish preliminary classification benchmarks to guide algorithm selection on Fashion-MNIST.

Proposed method

• Use 70,000 images (60,000 train, 10,000 test) across 10 classes with 6,000 training examples per class.
• Process Zalando front-look thumbnails through a conversion pipeline to 28x28 grayscale images with normalization steps.
• Evaluate a wide range of classifiers (DT, RF, SVM, KNN, Logistic Regression, MLP, etc.) with 5-fold shuffles to generate average test accuracies.
• Provide a public repository with extended benchmark results and experimental details.

Experimental results

Research questions

• RQ1 Can Fashion-MNIST reliably replace MNIST for benchmarking while increasing classification difficulty?
• RQ2 How do common ML algorithms perform on Fashion-MNIST compared to MNIST under similar data formats and splits?
• RQ3 What baseline performance ranges can guide future algorithm development on Fashion-MNIST?

Key findings

• Fashion-MNIST achieves a more challenging task than MNIST while maintaining MNIST-compatible data format and splits.
• A broad set of classifiers were evaluated, showing varying performance across families (tree-based, SVM, k-NN, logistic, neural nets).
• GradientBoosting, RandomForest, and SVC variants achieve high accuracies on Fashion-MNIST, illustrating strong performance potential with proper tuning.
• The dataset is designed to be easily substituted for MNIST in existing ML pipelines, enabling quick benchmarking adoption.