[Paper Review] Combining Weakly and Webly Supervised Learning for Classifying Food Images

Parneet Kaur, Karan Sikka+1 more|arXiv (Cornell University)|Dec 23, 2017

Image Retrieval and Classification Techniques20 references3 citations

TL;DR

This paper proposes a hybrid approach combining webly supervised learning with weakly supervised learning (WSL) to improve food image classification using minimal manual curation. By sequentially integrating a small amount of manually curated data into large-scale noisy web data and applying WSL to localize discriminative regions, the method achieves a top-1 accuracy of 76.2%, significantly outperforming models trained on curated-only data (63.3%) or web-only data (50.3%).

ABSTRACT

Food classification from images is a fine-grained classification problem. Manual curation of food images is cost, time and scalability prohibitive. On the other hand, web data is available freely but contains noise. In this paper, we address the problem of classifying food images with minimal data curation. We also tackle a key problems with food images from the web where they often have multiple cooccuring food types but are weakly labeled with a single label. We first demonstrate that by sequentially adding a few manually curated samples to a larger uncurated dataset from two web sources, the top-1 classification accuracy increases from 50.3% to 72.8%. To tackle the issue of weak labels, we augment the deep model with Weakly Supervised learning (WSL) that results in an increase in performance to 76.2%. Finally, we show some qualitative results to provide insights into the performance improvements using the proposed ideas.

Motivation & Objective

• Address the high cost and scalability limitations of manually curated food image datasets for fine-grained classification.
• Tackle the challenge of noisy, multi-food, and weakly labeled web data in food image classification.
• Improve classification accuracy by combining limited manually curated data with large-scale uncurated web data.
• Utilize weakly supervised learning (WSL) to localize discriminative regions and reduce confusion between visually similar food classes.
• Demonstrate that performance improves linearly with increasing amounts of curated data and that WSL enhances generalization and localization accuracy.

Proposed method

• Collect uncurated web data from search engines (e.g., Google Images) to form a large-scale, noisy training set.
• Sequentially integrate a small number of manually curated images into the web data to improve model performance incrementally.
• Apply weakly supervised learning (WSL) to deep neural networks using only image-level labels, enabling activation map generation for localization.
• Use class activation maps (CAMs) from WSL to identify and focus on discriminative regions of food items, even in images with multiple co-occurring foods.
• Train a deep learning model end-to-end on the combined dataset (curated + web) with WSL supervision to improve generalization and reduce cross-category noise effects.
• Evaluate performance using top-1 and top-5 accuracy on the UEC256 test set, and analyze localization maps for qualitative insights.

Experimental results

Research questions

• RQ1 Can combining a small amount of manually curated data with large-scale web data improve food image classification accuracy compared to using either source alone?
• RQ2 How does the integration of weakly supervised learning (WSL) enhance model performance when training on noisy, multi-food, and weakly labeled web data?
• RQ3 To what extent does WSL improve localization of discriminative food regions, especially in cases of visual similarity between classes?
• RQ4 How do data distribution shifts between training and test sets (e.g., different cuisines) affect model generalization and misclassification rates?
• RQ5 What are the failure modes of WSL in multi-food scenarios, and how do they impact top-1 vs. top-5 prediction performance?

Key findings

• Top-1 accuracy increased from 50.3% using only web data to 72.8% by adding a small amount of manually curated data, demonstrating linear performance gain with curation.
• Further integration of weakly supervised learning (WSL) improved top-1 accuracy to 76.2%, outperforming the 63.3% accuracy of a model trained solely on curated data.
• WSL enabled approximate localization of food items via activation maps, helping disambiguate visually similar classes such as miso soup and chocolate cake.
• Top-5 accuracy reached 90.8%, indicating that even when the top-1 prediction was incorrect, the correct label was often in the top-5 predictions, especially in cases of partial occlusion or co-occurring foods.
• The model sometimes localized co-occurring food items together (e.g., chicken and rice) or focused on non-target items (e.g., sauce instead of gyoza), revealing limitations of standard WSL in multi-object scenarios.
• Misclassifications were frequently caused by distribution shifts (e.g., Japanese cuisine in test set not seen in training) or inter-class similarity (e.g., french fries vs. fish and chips), which WSL helped mitigate through localization.