[Paper Review] Dual Adaptive Transformations for Weakly Supervised Point Cloud Segmentation

Zhonghua Wu, Yicheng Wu+3 more|arXiv (Cornell University)|Jul 19, 2022

3D Shape Modeling and Analysis5 citations

TL;DR

This paper proposes Dual Adaptive Transformations (DAT), a novel weakly supervised point cloud segmentation framework that enforces consistency through adaptive local and regional perturbations to improve generalization from sparse annotations. By integrating Local Adaptive Perturbation (LAP) and Regional Adaptive Deformation (RAD) modules, DAT achieves state-of-the-art performance, outperforming prior methods by 6.5% mIoU on S3DIS under the 'One Thing One Click' setting.

ABSTRACT

Weakly supervised point cloud segmentation, i.e. semantically segmenting a point cloud with only a few labeled points in the whole 3D scene, is highly desirable due to the heavy burden of collecting abundant dense annotations for the model training. However, existing methods remain challenging to accurately segment 3D point clouds since limited annotated data may lead to insufficient guidance for label propagation to unlabeled data. Considering the smoothness-based methods have achieved promising progress, in this paper, we advocate applying the consistency constraint under various perturbations to effectively regularize unlabeled 3D points. Specifically, we propose a novel DAT ( extbf{D}ual extbf{A}daptive extbf{T}ransformations) model for weakly supervised point cloud segmentation, where the dual adaptive transformations are performed via an adversarial strategy at both point-level and region-level, aiming at enforcing the local and structural smoothness constraints on 3D point clouds. We evaluate our proposed DAT model with two popular backbones on the large-scale S3DIS and ScanNet-V2 datasets. Extensive experiments demonstrate that our model can effectively leverage the unlabeled 3D points and achieve significant performance gains on both datasets, setting new state-of-the-art performance for weakly supervised point cloud segmentation.

Motivation & Objective

• To address the challenge of training accurate 3D point cloud segmentation models with only a few labeled points.
• To improve label propagation and model generalization in weakly supervised settings by leveraging unlabeled data more effectively.
• To develop a robust, consistent training strategy that enforces smoothness at both point-level and region-level through adaptive transformations.
• To achieve state-of-the-art performance on large-scale benchmarks like S3DIS and ScanNet-v2 with minimal supervision.

Proposed method

• Proposes a Dual Adaptive Transformation (DAT) framework that applies consistency constraints via adaptive perturbations at both point-level and region-level.
• Introduces the Local Adaptive Perturbation (LAP) module, which generates adaptive coordinate and class-aware feature perturbations based on online-estimated class distributions.
• Employs a Regional Adaptive Deformation (RAD) module that applies adaptive affine transformations—such as shifting, scaling, and rotation—on superpoint regions to enforce structural consistency.
• Uses an adversarial strategy to train the transformation modules, ensuring that predictions remain consistent under diverse, adaptive perturbations.
• Applies the method end-to-end with standard point cloud backbones (e.g., KPConv, PointNet++) and integrates seamlessly with existing weakly supervised frameworks.
• Leverages pseudo-labeling and iterative refinement, where consistent predictions across perturbed views guide the learning of unlabeled points.

Experimental results

Research questions

• RQ1 Can adaptive, consistency-based data augmentation improve weakly supervised 3D point cloud segmentation when only sparse annotations are available?
• RQ2 How can local and regional smoothness constraints be effectively modeled using adaptive transformations tailored to point cloud geometry and feature distributions?
• RQ3 Can the proposed dual-level transformation strategy generalize across different point cloud architectures and benchmarks?
• RQ4 To what extent can the method improve existing state-of-the-art models like 1T1C through plug-and-play integration?

Key findings

• On the S3DIS dataset under the 'One Thing One Click' setting, DAT achieves a 6.5% mIoU improvement over the previous SOTA method 1T1C, reaching 62.3% mIoU.
• On the ScanNet-v2 validation set, DAT improves the mIoU by 3.0% over 1T1C, achieving 64.4% mIoU when built upon the 1T1C framework.
• The proposed method improves baseline models by 3.2% and 3.9% mIoU on the ScanNet-v2 test and validation sets, respectively, demonstrating strong generalization.
• When applied to a voxel-based backbone (MinkowskiNet), DAT improves mIoU by 5.9% and 3.2% under OTOC and OTTC settings, confirming its generalizability across architectures.
• The ablation study confirms that both LAP and RAD modules contribute significantly to performance, with class-aware feature perturbations enhancing local consistency.
• Qualitative results show that DAT preserves object structures and produces accurate, coherent segmentation even with only 20 labeled points.