[Paper Review] FoodX-251: A Dataset for Fine-grained Food Classification

Parneet Kaur, Karan Sikka+3 more|arXiv (Cornell University)|Jul 14, 2019

Image Retrieval and Classification Techniques24 references65 citations

TL;DR

This paper introduces FoodX-251, a large-scale web-sourced dataset with 251 fine-grained food categories and 158k images, plus human-verified labels for validation and test sets, and provides baseline results using ResNet-101.

ABSTRACT

Food classification is a challenging problem due to the large number of categories, high visual similarity between different foods, as well as the lack of datasets for training state-of-the-art deep models. Solving this problem will require advances in both computer vision models as well as datasets for evaluating these models. In this paper we focus on the second aspect and introduce FoodX-251, a dataset of 251 fine-grained food categories with 158k images collected from the web. We use 118k images as a training set and provide human verified labels for 40k images that can be used for validation and testing. In this work, we outline the procedure of creating this dataset and provide relevant baselines with deep learning models. The FoodX-251 dataset has been used for organizing iFood-2019 challenge in the Fine-Grained Visual Categorization workshop (FGVC6 at CVPR 2019) and is available for download.

Motivation & Objective

• Motivate the need for large-scale, fine-grained food datasets to advance deep learning for food classification.
• Describe the creation and cleaning process for FoodX-251, including handling web-derived noise.
• Provide baseline results to benchmark future methods on a challenging, diverse food dataset.

Proposed method

• Collect 251 fine-grained food classes by expanding Food-101 through WordNet siblings and manual filtering.
• Download web images for each class and apply duplicate removal and noise filtering (cross-domain and cross-category).
• Hum​an verification on a random sample per class to produce clean validation and test splits (12k val, 28k test; 40k with verification available).
• Use a simple evaluation metric inspired by ILSVRC: top-3 label accuracy where error per image is 0 if any predicted label matches ground truth, else 1.
• Provide baseline deep learning results using ResNet-101 with two fine-tuning strategies (last-layer vs all-layers).
• Demonstrate baseline performance to motivate further research with FGVC-style food recognition challenges.

Experimental results

Research questions

• RQ1 Can a large-scale, fine-grained food dataset collected from web sources support training state-of-the-art deep learning models?
• RQ2 How do noise and ambiguity in web-sourced food images affect labeling and model performance?
• RQ3 What baseline performance do standard architectures achieve on FoodX-251, and how does fine-tuning scope affect results?

Key findings

• FoodX-251 comprises 251 classes and 158k images, with 118k training, 12k validation, and 28k test images; 40k images have human-verified labels.
• Baseline ResNet-101 results show a substantial gap between last-layer fine-tuning and full-network fine-tuning, with top-3 error dropping from 0.37 to 0.17 when fine-tuning all layers.
• Full-network fine-tuning yields a notable performance improvement over last-layer fine-tuning for this fine-grained task.
• The dataset was used to organize the iFood-2019 FGVC workshop challenge at CVPR 2019 (FGVC6).
• FoodX-251 provides a challenging benchmark with miscellaneous cuisine items and high intra-class similarity, suitable for evaluating fine-grained food classifiers.