[Paper Review] SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes

Zhili Ng, Haozhe Wang+3 more|arXiv (Cornell University)|Jul 14, 2023

Advanced Neural Network Applications4 citations

TL;DR

SynTable is a photorealistic, Python-based synthetic data generation pipeline built on NVIDIA's Isaac Sim Replicator Composer that automatically generates high-quality, fully-annotated datasets for unseen object amodal instance segmentation in cluttered tabletop scenes. By eliminating manual labeling and enabling rich metadata—such as amodal masks, occlusion order, and depth maps—SynTable significantly improves Sim-to-Real transfer performance, with UOAIS-Net achieving 87.0% F@.75 and 91.9% occlusion order accuracy on the OSD-Amodal benchmark when trained on SynTable-generated data.

ABSTRACT

In this work, we present SynTable, a unified and flexible Python-based dataset generator built using NVIDIA's Isaac Sim Replicator Composer for generating high-quality synthetic datasets for unseen object amodal instance segmentation of cluttered tabletop scenes. Our dataset generation tool can render complex 3D scenes containing object meshes, materials, textures, lighting, and backgrounds. Metadata, such as modal and amodal instance segmentation masks, object amodal RGBA instances, occlusion masks, depth maps, bounding boxes, and material properties can be automatically generated to annotate the scene according to the users' requirements. Our tool eliminates the need for manual labeling in the dataset generation process while ensuring the quality and accuracy of the dataset. In this work, we discuss our design goals, framework architecture, and the performance of our tool. We demonstrate the use of a sample dataset generated using SynTable for training a state-of-the-art model, UOAIS-Net. Our state-of-the-art results show significantly improved performance in Sim-to-Real transfer when evaluated on the OSD-Amodal dataset. We offer this tool as an open-source, easy-to-use, photorealistic dataset generator for advancing research in deep learning and synthetic data generation. The links to our source code, demonstration video, and sample dataset can be found in the supplementary materials.

Motivation & Objective

• To address the lack of large-scale, high-quality datasets for unseen object amodal instance segmentation (UOAIS) in cluttered tabletop scenes.
• To overcome the limitations of manual annotation, which is time-consuming, subjective, and prone to error for amodal masks.
• To reduce the Sim-to-Real domain gap by generating photorealistic synthetic data with accurate, automatic annotations.
• To provide a unified, flexible, and open-source tool for generating diverse, customizable synthetic datasets with rich metadata for deep learning in robotics.
• To enable improved performance in amodal completion and occlusion reasoning for robotic grasping systems through better training data.

Proposed method

• The pipeline uses NVIDIA’s Isaac Sim Replicator Composer to render complex 3D tabletop scenes with photorealistic textures, lighting, and object meshes.
• It automatically generates ground-truth annotations including amodal and visible instance segmentation masks, occlusion masks, depth maps, bounding boxes, and material properties.
• Users can customize scene composition, object types, and required annotations via a flexible Python-based interface.
• The framework integrates domain randomization and high-fidelity rendering to minimize domain shift and enhance Sim-to-Real generalization.
• A novel occlusion order adjacency matrix is generated to capture object-to-object occlusion relationships, enabling better reasoning in cluttered scenes.
• The pipeline supports end-to-end dataset generation with configurable output formats for training state-of-the-art models like UOAIS-Net.

Experimental results

Research questions

• RQ1 Can a synthetic data generation pipeline produce photorealistic, fully-annotated datasets for UOAIS without manual labeling?
• RQ2 To what extent does synthetic data from SynTable improve Sim-to-Real transfer performance for amodal instance segmentation models?
• RQ3 How does the inclusion of occlusion order annotations affect the performance of amodal segmentation models in cluttered scenes?
• RQ4 Can SynTable generate datasets that outperform existing synthetic benchmarks in terms of occlusion reasoning and mask prediction accuracy?
• RQ5 What is the impact of domain randomization and photorealism on the generalization of UOAIS-Net when trained on SynTable-generated data?

Key findings

• UOAIS-Net trained on SynTable-generated data achieved 87.0% F@.75 on the OSD-Amodal benchmark, a significant improvement over the 80.4% F@.75 when trained on the UOAIS-Sim dataset.
• The occlusion order accuracy ($ACC_{OO}$) of UOAIS-Net increased from 84.0% to 91.9% when fine-tuned on SynTable-Sim data, indicating superior occlusion reasoning.
• SynTable-generated data improved the boundary F-measure for amodal masks from 50.3% to 53.7% and for visible masks from 50.0% to 53.7% compared to UOAIS-Sim.
• The model trained on SynTable-Sim achieved 86.2% overlap F-measure for amodal masks and 80.1% for visible masks on the SynTable-Sim validation set, outperforming the UOAIS-Sim baseline.
• SynTable reduced the domain gap, enabling UOAIS-Net to generalize better to real-world scenes, as evidenced by the 6.6% absolute improvement in occlusion order accuracy.
• The pipeline successfully eliminated manual labeling while ensuring high annotation quality, as validated by consistent improvements across precision, recall, and F-measure metrics.