[Paper Review] Cold Case: The Lost MNIST Digits

Chhavi Yadav, Léon Bottou|arXiv (Cornell University)|May 25, 2019

Machine Learning and Data Classification10 references39 citations

TL;DR

The authors reconstruct the MNIST preprocessing pipeline to recover the lost 50,000 MNIST test digits and pair them with metadata, enabling controlled comparisons of classifier performance and model selection over repeated test-set use.

ABSTRACT

Although the popular MNIST dataset [LeCun et al., 1994] is derived from the NIST database [Grother and Hanaoka, 1995], the precise processing steps for this derivation have been lost to time. We propose a reconstruction that is accurate enough to serve as a replacement for the MNIST dataset, with insignificant changes in accuracy. We trace each MNIST digit to its NIST source and its rich metadata such as writer identifier, partition identifier, etc. We also reconstruct the complete MNIST test set with 60,000 samples instead of the usual 10,000. Since the balance 50,000 were never distributed, they enable us to investigate the impact of twenty-five years of MNIST experiments on the reported testing performances. Our results unambiguously confirm the trends observed by Recht et al. [2018, 2019]: although the misclassification rates are slightly off, classifier ordering and model selection remain broadly reliable. We attribute this phenomenon to the pairing benefits of comparing classifiers on the same digits.

Motivation & Objective

• Reconstruct the MNIST preprocessing steps from NIST to map each MNIST digit to its original NIST source and metadata.
• Rebuild the MNIST training set and recreate the full 60,000-sample test set, including the 50,000 lost test digits.
• Assess how well reconstructed data match official MNIST samples and study the impact of test-set reuse on performance trends.
• Investigate classifier performance across MNIST, QMNIST10K, and QMNIST50K using paired comparisons and rigorous confidence intervals.

Proposed method

• Iteratively refine an image reconstruction pipeline (QMNIST variants) to closely match MNIST digits using center-of-gravity centering, cropping, and a pixel-overlap resampling approach.
• Quantify reconstruction quality with L2 and L-infinity distances and alignment checks, including occasional one-pixel shifts.
• Train and evaluate multiple models (KNN, SVM, MLP, CNNs) on MNIST and QMNIST training sets, testing on MNIST, QMNIST10K, and QMNIST50K.
• Use Wald confidence intervals and paired difference tests to assess statistical significance and account for repeated test-set usage.

Experimental results

Research questions

• RQ1 Can the lost MNIST 50,000 test digits be reconstructed closely enough to serve as a valid test set replacement?
• RQ2 How does reusing a testing set across many models affect reported performance and model selection, and can paired comparisons mitigate these effects?
• RQ3 Do classifier rankings on MNIST persist when evaluated on reconstructed equivalents (QMNIST) and the reconstructed 50k test digits?
• RQ4 What systematic artifacts exist in MNIST preprocessing (centering, resampling, antialiasing) and how do they influence downstream performance?
• RQ5 How do modern models (KNN, SVM, MLP, VGG-11, ResNet-18, TF-KR MNIST models) rank across MNIST and its reconstructions?

Key findings

• The reconstructed 60k training and 60k test sets closely resemble official MNIST samples, with small misalignment in about 0.25% of QMNIST training images due to centering shifts.
• Training on MNIST vs QMNIST yields comparable performance on MNIST test and QMNIST10K, but modest degradation on QMNIST50K (the reconstructed lost digits).
• Best performing MNIST models generally retain strong performance order on QMNIST50K, indicating classifier ranking is preserved despite reconstruction imperfections.
• Confidence-interval and paired-difference analysis confirms the standard test-set rot problem exists but is less severe than feared, with pairing aiding model selection.
• Across KNN, SVM, MLP, and CNN variants, the MNIST rankings closely predict QMNIST50K performance, though absolute error rates are slightly higher on the reconstructed 50k set.