[논문 리뷰] Identifying Object States in Cooking-Related Images

Ahmad Babaeian Jelodar, Sirajus Salekin외 1명|arXiv (Cornell University)|2018. 05. 17.

Image Retrieval and Classification Techniques참고 문헌 21인용 수 47

한 줄 요약

본 논문은 요리 이미지에서 물체 상태를 식별하는 문제를 정의하고 이를 해결하며, 17개 물체에 대해 11상태 데이터셋을 구축하고, 물체별 미세조정을 포함한 ResNet 기반 전이 학습 모델을 제시하여 강력한 상태 분류 성능을 달성하고 Imagenet 부분집합에서 상태 표기를 가능하게 한다.

ABSTRACT

Understanding object states is as important as object recognition for robotic task planning and manipulation. To our knowledge, this paper explicitly introduces and addresses the state identification problem in cooking related images for the first time. In this paper, objects and ingredients in cooking videos are explored and the most frequent objects are analyzed. Eleven states from the most frequent cooking objects are examined and a dataset of images containing those objects and their states is created. As a solution to the state identification problem, a Resnet based deep model is proposed. The model is initialized with Imagenet weights and trained on the dataset of eleven classes. The trained state identification model is evaluated on a subset of the Imagenet dataset and state labels are provided using a combination of the model with manual checking. Moreover, an individual model is fine-tuned for each object in the dataset using the weights from the initially trained model and object-specific images, where significant improvement is demonstrated.

연구 동기 및 목표

• Define the state identification problem for cooking objects to aid fine-grained activity understanding and robot manipulation.
• Create a labeled dataset of cooking objects and 11 states.
• Propose a ResNet-based transfer learning architecture for state classification.
• Show that per-object fine-tuning improves state-label accuracy.
• Demonstrate state labeling on a subset of Imagenet for cooking-related objects.

제안 방법

• Construct a state dataset with 17 cooking objects and 11 states (whole, peeled, floured, sliced, diced, grated, julienne, juice, creamy, mixed, other).
• Develop a ResNet-based network with a 1x1 conv layer, two conv layers, and global average pooling, followed by an 11-class softmax for state classification.
• Use Imagenet-pretrained weights, freezing the base initially and then fine-tuning the entire network.
• Perform object-specific fine-tuning by replacing the final layer with a softmax sized to the number of states for each object and fine-tuning in four stages.
• Evaluate with online augmentation, L2 regularization, and batch normalization; employ a voting ensemble with two additional models.

실험 결과

연구 질문

• RQ1 What are the distinct states objects in cooking images can occupy and how can they be reliably classified?
• RQ2 Does object-specific fine-tuning improve state identification accuracy over a single, global model?
• RQ3 How well does the state-identification model generalize to images from the Imagenet dataset containing cooking-related objects?
• RQ4 Can the model provide state labels for objects in large-scale image datasets beyond the custom dataset?

주요 결과

• State identification accuracy on the full state dataset without object-specific fine-tuning is 80.4% Top-1 (81.4% average per class) and 91.5% Top-2, rising to 82% with voting ensemble.
• Object-specific fine-tuning yields higher performance, with average Top-1 86.9% and Top-2 88.3% across objects (excluding dough in that particular experiment).
• Imagenet subset evaluation achieves 78.5% Top-1, 89.6% Top-2, and 94.5% Top-3 accuracy when labeling 16 cooking-related object categories with 11 state classes.
• Per-object fine-tuning significantly improves state identification accuracy over the global model (examples show substantial gains across several objects, e.g., mushroom 95.6% Top-1; carrot 96% Top-1 in some settings).
• An Imagenet-based labeling pass assigned state labels to 800 images across 16 object categories, with average 78.5% Top-1, 89.6% Top-2, and 94.5% Top-3.
• The paper notes state misclassification often arises from multi-state or ambiguous images, suggesting future work in multi-state detection and video tracking.