[Paper Review] ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes

Dina Bashkirova, Mohamed S. Abdelfattah+8 more|arXiv (Cornell University)|Jun 4, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper introduces ZeroWaste, the largest publicly available in-the-wild dataset for industrial waste detection, featuring densely annotated frames of highly deformable, translucent, and cluttered recyclable objects from real Materials Recovery Facilities. The dataset enables benchmarking of fully, semi-, and weakly supervised segmentation models, with results showing that state-of-the-art methods struggle with class imbalance and context-poor scenes, highlighting the need for label-efficient learning in real-world robotics and recycling applications.

ABSTRACT

Less than 35% of recyclable waste is being actually recycled in the US, which leads to increased soil and sea pollution and is one of the major concerns of environmental researchers as well as the common public. At the heart of the problem are the inefficiencies of the waste sorting process (separating paper, plastic, metal, glass, etc.) due to the extremely complex and cluttered nature of the waste stream. Recyclable waste detection poses a unique computer vision challenge as it requires detection of highly deformable and often translucent objects in cluttered scenes without the kind of context information usually present in human-centric datasets. This challenging computer vision task currently lacks suitable datasets or methods in the available literature. In this paper, we take a step towards computer-aided waste detection and present the first in-the-wild industrial-grade waste detection and segmentation dataset, ZeroWaste. We believe that ZeroWaste will catalyze research in object detection and semantic segmentation in extreme clutter as well as applications in the recycling domain. Our project page can be found at http://ai.bu.edu/zerowaste/.

Motivation & Objective

• To address the lack of high-quality, real-world datasets for detecting and segmenting highly deformable, translucent, and cluttered recyclable objects in industrial waste streams.
• To enable research in fully, semi-, and weakly supervised learning for object detection and semantic segmentation in extreme clutter and class-imbalanced scenarios.
• To support the development of data-efficient, robust computer vision models applicable to real-world recycling automation and safer MRF operations.
• To provide a benchmark for evaluating label-efficient learning techniques in industrial settings with limited annotated data.

Proposed method

• The ZeroWaste dataset is collected from real conveyor belts in Materials Recovery Facilities, containing fully annotated frames (ZeroWaste-f) and unlabeled frames (ZeroWaste-s) for semi-supervised learning.
• The dataset includes four foreground classes: soft plastic, rigid plastic, cardboard, and metal, with paper and conveyor belt as background.
• A data-augmentation strategy introduces objects from the TACO dataset into ZeroWaste to mitigate class imbalance, creating the ZeroWaste-Aug variant.
• Baseline models include DeepLabV3+ for fully supervised learning, ReCo and CCT for semi-supervised learning using unlabeled data, and CAM-based methods (including Puzzle-CAM and EPS) for weakly supervised segmentation.
• Pseudo-labels for weakly supervised learning are generated via Class Activation Maps (CAMs) with thresholding at 0.45 and enhanced object coverage using tile-based matching.
• Evaluation is conducted across fully, semi-, and weakly supervised settings, with performance measured via pixel accuracy and confusion matrices.

Experimental results

Research questions

• RQ1 Can state-of-the-art fully supervised segmentation models generalize to the extreme clutter and deformability found in real industrial waste streams?
• RQ2 To what extent can unlabeled data from the ZeroWaste-s subset improve performance in semi-supervised learning, and which inductive biases are effective in this context?
• RQ3 Can weakly supervised methods using only image-level labels (e.g., before/after labels) generate meaningful pseudo-segmentation masks for training robust models?
• RQ4 How effective is instance-level data augmentation (e.g., ZeroWaste-Aug) in reducing performance gaps for rare classes like metal in imbalanced datasets?

Key findings

• Fully supervised DeepLabV3+ achieves only 58% accuracy in distinguishing cardboard from paper, indicating significant challenges in fine-grained segmentation.
• Rigid plastic objects are frequently misclassified as soft plastic, highlighting difficulties in texture and shape generalization.
• Metal objects are often segmented as background due to their low frequency in the original dataset, underscoring the impact of class imbalance.
• Data augmentation with ZeroWaste-Aug improves pixel accuracy for the rare metal class on the original test split, but degrades performance on rigid plastic, showing trade-offs in augmentation strategies.
• Semi-supervised learning with ReCo underperforms when using unlabeled data, suggesting that ReCo’s context-based inductive bias is ineffective in low-context, cluttered scenes.
• CCT shows slight performance gains with unlabeled data, indicating that contrastive learning is better suited to the domain’s visual complexity than context-aware models.