[Paper Review] EMNIST: an extension of MNIST to handwritten letters

Gregory Cohen, Saeed Afshar+2 more|arXiv (Cornell University)|Feb 17, 2017

Advanced Neural Network Applications10 references459 citations

TL;DR

EMNIST creates multiple EMNIST datasets by extending MNIST with NIST Special Database 19 characters, converting to 28x28 images, and provides benchmark results using OPIUM-based classifiers.

ABSTRACT

The MNIST dataset has become a standard benchmark for learning, classification and computer vision systems. Contributing to its widespread adoption are the understandable and intuitive nature of the task, its relatively small size and storage requirements and the accessibility and ease-of-use of the database itself. The MNIST database was derived from a larger dataset known as the NIST Special Database 19 which contains digits, uppercase and lowercase handwritten letters. This paper introduces a variant of the full NIST dataset, which we have called Extended MNIST (EMNIST), which follows the same conversion paradigm used to create the MNIST dataset. The result is a set of datasets that constitute a more challenging classification tasks involving letters and digits, and that shares the same image structure and parameters as the original MNIST task, allowing for direct compatibility with all existing classifiers and systems. Benchmark results are presented along with a validation of the conversion process through the comparison of the classification results on converted NIST digits and the MNIST digits.

Motivation & Objective

• Motivate the need for a broader, well-defined benchmark combining digits and letters.
• Describe a conversion process to create MNIST-compatible 28x28 images from NIST 128x128 data.
• Introduce six EMNIST datasets (By_Class, By_Merge, Balanced, Letters, Digits, MNIST) with train/validation/test splits.
• Provide baseline classification results using OPIUM-based networks to validate dataset usefulness and conversion quality.

Proposed method

• Convert NIST Special Database 19 images to 28x28 grayscale format matching MNIST using Gaussian blurring, ROI extraction, centering, padding, and bicubic down-sampling.
• Split data following MNIST methodology by combining training/testing into a random split to create training, testing, and validation (where applicable).
• Apply an Online Pseudo-Inverse Update Method (OPIUM) based three-layer ELM network to establish baselines across datasets.
• Train a linear classifier as a baseline using a pseudo-inverse solution for networks without hidden layers.
• Compare performance against the original MNIST to validate the conversion and dataset structure.

Experimental results

Research questions

• RQ1 Can EMNIST datasets offer a more challenging and realistic benchmark than MNIST by including letters with digits?
• RQ2 Does the conversion pipeline preserve separability and enable drop-in compatibility with MNIST-based systems?
• RQ3 How do OPIUM-based and linear classifiers perform across EMNIST By_Class, By_Merge, Balanced, Letters, Digits, and MNIST datasets?
• RQ4 What insights into uppercase/lowercase confusion and digit-letter classification emerge from the confusion matrices and per-class results?

Key findings

• EMNIST Balanced achieved 78.02% ±0.92% with 10,000 hidden neurons (OPIUM) vs 50.93% linear baseline.
• EMNIST By_Merge achieved 72.57% ±1.18% (OPIUM) vs 50.51% linear baseline.
• EMNIST By_Class achieved 69.71% ±1.47% (OPIUM) vs 51.80% linear baseline.
• EMNIST Letters achieved 85.15% ±0.12% (OPIUM) vs 55.78% linear baseline.
• EMNIST MNIST achieved 85.11% (OPIUM) with the same network structure used for digits; higher than original MNIST in this baseline setup.
• Digits-only and MNIST-compatible subsets show high accuracy with the new conversion, indicating improved separability over standard MNIST.