[Paper Review] Identifying Object States in Cooking-Related Images

Ahmad Babaeian Jelodar, Sirajus Salekin+1 more|arXiv (Cornell University)|May 17, 2018

Image Retrieval and Classification Techniques21 references47 citations

TL;DR

The paper defines and tackles the problem of identifying object states in cooking images, builds a 11-state dataset for 17 objects, and presents a ResNet-based transfer learning model with per-object fine-tuning, achieving strong state-classification performance and enabling state labeling on Imagenet subsets.

ABSTRACT

Understanding object states is as important as object recognition for robotic task planning and manipulation. To our knowledge, this paper explicitly introduces and addresses the state identification problem in cooking related images for the first time. In this paper, objects and ingredients in cooking videos are explored and the most frequent objects are analyzed. Eleven states from the most frequent cooking objects are examined and a dataset of images containing those objects and their states is created. As a solution to the state identification problem, a Resnet based deep model is proposed. The model is initialized with Imagenet weights and trained on the dataset of eleven classes. The trained state identification model is evaluated on a subset of the Imagenet dataset and state labels are provided using a combination of the model with manual checking. Moreover, an individual model is fine-tuned for each object in the dataset using the weights from the initially trained model and object-specific images, where significant improvement is demonstrated.

Motivation & Objective

• Define the state identification problem for cooking objects to aid fine-grained activity understanding and robot manipulation.
• Create a labeled dataset of cooking objects and 11 states.
• Propose a ResNet-based transfer learning architecture for state classification.
• Show that per-object fine-tuning improves state-label accuracy.
• Demonstrate state labeling on a subset of Imagenet for cooking-related objects.

Proposed method

• Construct a state dataset with 17 cooking objects and 11 states (whole, peeled, floured, sliced, diced, grated, julienne, juice, creamy, mixed, other).
• Develop a ResNet-based network with a 1x1 conv layer, two conv layers, and global average pooling, followed by an 11-class softmax for state classification.
• Use Imagenet-pretrained weights, freezing the base initially and then fine-tuning the entire network.
• Perform object-specific fine-tuning by replacing the final layer with a softmax sized to the number of states for each object and fine-tuning in four stages.
• Evaluate with online augmentation, L2 regularization, and batch normalization; employ a voting ensemble with two additional models.

Experimental results

Research questions

• RQ1 What are the distinct states objects in cooking images can occupy and how can they be reliably classified?
• RQ2 Does object-specific fine-tuning improve state identification accuracy over a single, global model?
• RQ3 How well does the state-identification model generalize to images from the Imagenet dataset containing cooking-related objects?
• RQ4 Can the model provide state labels for objects in large-scale image datasets beyond the custom dataset?

Key findings

• State identification accuracy on the full state dataset without object-specific fine-tuning is 80.4% Top-1 (81.4% average per class) and 91.5% Top-2, rising to 82% with voting ensemble.
• Object-specific fine-tuning yields higher performance, with average Top-1 86.9% and Top-2 88.3% across objects (excluding dough in that particular experiment).
• Imagenet subset evaluation achieves 78.5% Top-1, 89.6% Top-2, and 94.5% Top-3 accuracy when labeling 16 cooking-related object categories with 11 state classes.
• Per-object fine-tuning significantly improves state identification accuracy over the global model (examples show substantial gains across several objects, e.g., mushroom 95.6% Top-1; carrot 96% Top-1 in some settings).
• An Imagenet-based labeling pass assigned state labels to 800 images across 16 object categories, with average 78.5% Top-1, 89.6% Top-2, and 94.5% Top-3.
• The paper notes state misclassification often arises from multi-state or ambiguous images, suggesting future work in multi-state detection and video tracking.