[Paper Review] Tuned Inception V3 for Recognizing States of Cooking Ingredients

Kin Ng|arXiv (Cornell University)|May 5, 2019

Robot Manipulation and Learning23 references4 citations

TL;DR

This paper proposes a fine-tuned Inception V3 model for recognizing 11 distinct states of cooking ingredients (e.g., peeled, diced, whole) using transfer learning on a custom dataset. By freezing early layers and fine-tuning the top layers with SGD, the model achieves 69.4% accuracy on unseen test data, demonstrating feasibility for robotic cooking applications.

ABSTRACT

Cooking is a task that must be performed in a daily basis, and thus it is an activity that many people take for granted. For humans preparing a meal comes naturally, but for robots even preparing a simple sandwich results in an extremely difficult task. In robotics, designing kitchen robots is complicated since cooking relies on a variety of physical interactions that are dependent on different conditions such as changes in the environment, proper execution of sequential instructions, along with motions, and detection of the different states in which cooking-ingredients can be in for their correct grasping and manipulation. In this paper, we focus on the challenge of state recognition and propose a fine tuned convolutional neural network that makes use of transfer learning by reusing the Inception V3 pre-trained model. The model is trained and validated on a cooking dataset consisting of eleven states (e.g. peeled, diced, whole, etc.). The work presented on this paper could provide insight into finding a potential solution to the problem.

Motivation & Objective

• Address the challenge of robotic cooking by enabling accurate recognition of ingredient states, a prerequisite for proper grasping and manipulation.
• Overcome the difficulty of state recognition in dynamic kitchen environments where ingredients undergo multiple transformations.
• Leverage transfer learning to adapt a pre-trained Inception V3 model to a specialized cooking ingredient state classification task.
• Provide a foundational solution for robotic systems to understand and respond to the physical condition of ingredients during meal preparation.
• Contribute to the broader goal of enabling robots to perform complex, real-world daily tasks such as cooking autonomously.

Proposed method

• Utilized transfer learning by fine-tuning the pre-trained Inception V3 architecture on a custom dataset of cooking ingredient states.
• Split training into two stages: first, training only the randomly initialized top layers with a high initial learning rate; second, fine-tuning the entire network with a low learning rate.
• Applied data augmentation techniques including random horizontal flipping and color jittering to improve generalization.
• Selected SGD with a low learning rate (0.00001) and momentum (0.9) for the fine-tuning phase to ensure stable, small weight updates.
• Experimented with freezing different numbers of initial layers, finding that freezing up to layer 249 (i.e., the top two trainable blocks) yielded optimal performance.
• Used a custom dataset of 11 states, including ambiguous and multi-state examples, to evaluate model robustness and generalization.

Experimental results

Research questions

• RQ1 Can a fine-tuned Inception V3 model achieve reliable classification of 11 distinct cooking ingredient states using transfer learning?
• RQ2 How does freezing different numbers of initial layers affect model performance and convergence during fine-tuning?
• RQ3 What impact do data ambiguity, multi-state labeling, and the 'other' category have on classification accuracy?
• RQ4 Can a transfer learning approach with SGD fine-tuning outperform end-to-end training from scratch in this specialized cooking state recognition task?
• RQ5 To what extent do data quality issues such as mislabeled or ambiguous images limit model performance?

Key findings

• The model achieved 72% accuracy on the validation set after full training, indicating strong learning on the training distribution.
• On unseen test data, the model reached 69.4% accuracy, demonstrating generalization capability despite dataset limitations.
• Freezing up to the 249th layer (top two Inception blocks) yielded the best performance, with validation loss of 0.8192 and accuracy of 72.86%.
• Freezing more layers than this (e.g., up to layer 229 or 197) reduced accuracy and increased training time, suggesting over-freezing harms performance.
• The 'other' class and ambiguous/multi-state images were identified as key contributors to misclassification, particularly in challenging test cases.
• Visual analysis revealed that similar-looking images with different labels (e.g., 'creamy' vs. 'other') and multi-state images (e.g., both diced and floured) significantly impacted model reliability.