[Paper Review] Acquire, Augment, Segment & Enjoy: Weakly Supervised Instance Segmentation of Supermarket Products

Patrick Follmann, Bertram Drost+1 more|arXiv (Cornell University)|Jul 5, 2018

Industrial Vision Systems and Defect Detection18 references3 citations

TL;DR

This paper proposes a weakly supervised method for instance segmentation of supermarket products using only image-level class labels, leveraging simple turntable-based acquisition and advanced data augmentation to generate high-quality training data with minimal manual annotation. Despite a domain shift, the approach achieves 68.9% mean average precision, outperforming expectations for weak supervision and approaching 80.1% of a fully supervised baseline.

ABSTRACT

Grocery stores have thousands of products that are usually identified using barcodes with a human in the loop. For automated checkout systems, it is necessary to count and classify the groceries efficiently and robustly. One possibility is to use a deep learning algorithm for instance-aware semantic segmentation. Such methods achieve high accuracies but require a large amount of annotated training data. We propose a system to generate the training annotations in a weakly supervised manner, drastically reducing the labeling effort. We assume that for each training image, only the object class is known. The system automatically segments the corresponding object from the background. The obtained training data is augmented to simulate variations similar to those seen in real-world setups.

Motivation & Objective

• Reduce the high labeling cost of instance segmentation in industrial applications like automated checkout systems.
• Enable training of deep learning models with minimal manual annotation by using only image-level class labels.
• Address domain shift between simple training images and complex real-world test scenes through tailored data augmentation.
• Demonstrate that weakly supervised learning can achieve competitive performance compared to fully supervised baselines.
• Develop a scalable, low-effort pipeline for generating training data for specific product categories in controlled environments.

Proposed method

• Acquire individual product images on a turntable with a single class label per image, enabling automatic foreground segmentation via basic image processing.
• Use Otsu's thresholding and morphological operations to extract object masks from images with homogeneous backgrounds.
• Apply data augmentation techniques including random cropping, scaling, rotation, and color jittering to simulate real-world variations.
• Introduce two novel augmentation stages: lighting variation modeling and occlusion simulation to address reflections and overlapping objects.
• Train instance segmentation models (e.g., Mask R-CNN) on the weakly supervised, augmented training data.
• Evaluate performance on the D2S dataset, comparing weakly supervised models to fully supervised baselines under domain shift.

Experimental results

Research questions

• RQ1 Can instance segmentation models be trained effectively with only image-level class labels and no bounding boxes or pixel masks?
• RQ2 How effective is automated mask generation from simple turntable acquisitions in producing usable training data?
• RQ3 To what extent can data augmentation mitigate domain shift between controlled training images and complex real-world test scenes?
• RQ4 Does weakly supervised learning achieve performance competitive with fully supervised methods in industrial instance segmentation tasks?
• RQ5 How does the number of augmented training images affect model generalization and performance?

Key findings

• The weakly supervised method achieves a mean average precision (mAP) of 68.9% on the D2S test set, compared to 80.1% for a fully supervised baseline under domain shift.
• Despite using only class-level labels and simple acquisition, the automatically generated masks yield high-quality training data suitable for deep learning.
• The proposed lighting variation and occlusion augmentation stages significantly improve robustness to real-world challenges like reflections and overlapping products.
• Increasing the number of augmented images beyond a certain point does not improve performance, suggesting that data quality and realism matter more than sheer quantity.
• Data augmentation remains beneficial even when the number of labeled training images is large, indicating its role in generalization beyond just data expansion.
• The method enables competitive instance segmentation performance with minimal labeling effort, making it viable for industrial deployment in supermarket automation.