[Paper Review] Synthetic-to-real Composite Semantic Segmentation in Additive Manufacturing

Aliaksei Petsiuk, Harnoor Singh+2 more|arXiv (Cornell University)|Oct 14, 2022

Additive Manufacturing and 3D Printing Technologies4 citations

TL;DR

This paper proposes a synthetic-to-real domain adaptation framework for semantic segmentation in 3D printing using physics-based rendering and image-to-image style transfer. By applying real-to-synthetic style transfer before domain adaptation, the method improves mIoU on real 3D printing images from 61.10% to 75.19% when using a model trained exclusively on synthetic data, significantly reducing the synthetic-to-real domain gap.

ABSTRACT

The application of computer vision and machine learning methods in the field of additive manufacturing (AM) for semantic segmentation of the structural elements of 3-D printed products will improve real-time failure analysis systems and can potentially reduce the number of defects by enabling in situ corrections. This work demonstrates the possibilities of using physics-based rendering for labeled image dataset generation, as well as image-to-image translation capabilities to improve the accuracy of real image segmentation for AM systems. Multi-class semantic segmentation experiments were carried out based on the U-Net model and cycle generative adversarial network. The test results demonstrated the capacity of detecting such structural elements of 3-D printed parts as a top layer, infill, shell, and support. A basis for further segmentation system enhancement by utilizing image-to-image style transfer and domain adaptation technologies was also developed. The results indicate that using style transfer as a precursor to domain adaptation can significantly improve real 3-D printing image segmentation in situations where a model trained on synthetic data is the only tool available. The mean intersection over union (mIoU) scores for synthetic test datasets included 94.90% for the entire 3-D printed part, 73.33% for the top layer, 78.93% for the infill, 55.31% for the shell, and 69.45% for supports.

Motivation & Objective

• To address the challenge of high failure rates in 3D printing (up to 20% for beginners) by enabling real-time in situ defect detection.
• To reduce reliance on costly, time-consuming real-world annotation by leveraging synthetic data generation via physics-based rendering.
• To bridge the synthetic-to-real domain gap in 3D printing image segmentation using image-to-image translation and domain adaptation techniques.
• To develop a camera-agnostic, markerless monitoring system for 3D printing that reduces calibration and positioning requirements.
• To enable scalable, low-cost visual monitoring of 3D printed parts using a single camera and transferable deep learning models.

Proposed method

• Generated photorealistic synthetic training data using Blender’s physics-based rendering engine from G-code files of 3D printed parts.
• Trained a U-Net-based multi-class semantic segmentation model on synthetic images to identify top layer, infill, shell, and support structures.
• Applied real-to-synthetic image-to-image style transfer (using CycleGAN) to transform real 3D printing images to match the style of synthetic data.
• Used the style-transferred real images as input to the synthetic-trained model, effectively performing domain adaptation without real annotations.
• Evaluated performance using mean Intersection over Union (mIoU) on both synthetic and real test sets.
• Employed t-SNE visualization to analyze domain shift and validate the effectiveness of style transfer in reducing domain discrepancy.

Experimental results

Research questions

• RQ1 Can physics-based synthetic data generation effectively train a semantic segmentation model for 3D printed parts?
• RQ2 To what extent does real-to-synthetic image style transfer reduce the domain gap between synthetic and real 3D printing images?
• RQ3 How does style transfer as a preprocessing step improve segmentation accuracy when fine-tuning is not feasible?
• RQ4 What is the impact of domain adaptation via style transfer on mIoU scores for real-world 3D printing images?
• RQ5 Can a model trained solely on synthetic data achieve acceptable performance on real images after style transfer?

Key findings

• The model achieved a high mIoU of 94.90% on synthetic test data for the entire 3D printed part, indicating strong generalization on synthetic data.
• For real 3D printing images, mIoU improved from 61.10% to 75.19% when using real images that were first transformed via real-to-synthetic style transfer.
• The method demonstrated that style transfer as a precursor to domain adaptation significantly enhances segmentation performance in low-data regimes where real annotations are unavailable.
• The top layer (73.33% mIoU) and infill (78.93% mIoU) showed strong segmentation performance on synthetic data, while shell (55.31%) and supports (69.45%) were more challenging.
• Visual analysis confirmed that style transfer reduced distracting visual artifacts such as filament stringing and reflections, improving segmentation consistency.
• t-SNE projections validated that style transfer effectively reduced domain shift, aligning real image features closer to the synthetic distribution.