[Paper Review] Self-Supervised Learning for Domain Adaptation on Point-Clouds

Idan Achituve, Haggai Maron+1 more|arXiv (Cornell University)|Mar 29, 2020

Human Pose and Action RecognitionComputer Science69 references33 citations

TL;DR

The paper introduces Deformation Reconstruction (DefRec), a self-supervised learning task for domain adaptation on point clouds, plus Point Cloud Mixup (PCM), achieving state-of-the-art results on PointDA-10 and a new PointSegDA benchmark.

ABSTRACT

Self-supervised learning (SSL) is a technique for learning useful representations from unlabeled data. It has been applied effectively to domain adaptation (DA) on images and videos. It is still unknown if and how it can be leveraged for domain adaptation in 3D perception problems. Here we describe the first study of SSL for DA on point clouds. We introduce a new family of pretext tasks, Deformation Reconstruction, inspired by the deformations encountered in sim-to-real transformations. In addition, we propose a novel training procedure for labeled point cloud data motivated by the MixUp method called Point cloud Mixup (PCM). Evaluations on domain adaptations datasets for classification and segmentation, demonstrate a large improvement over existing and baseline methods.

Motivation & Objective

• Motivate and understand how self-supervised learning (SSL) can aid domain adaptation (DA) for 3D point clouds.
• Develop a family of SSL pretext tasks (DefRec) tailored to common sim-to-real deformations in point clouds.
• Propose a PCM (Point Cloud Mixup) method to augment labeled data for the source domain.
• Demonstrate through classification and segmentation benchmarks that DefRec and PCM improve DA performance over baselines.
• Provide analysis on deformation types, ablations, and practical guidelines for DefRec design.

Proposed method

• Propose a multi-task architecture with a shared point-cloud encoder Φ and two heads: h_sup for supervised source-domain task and h_SSL for SSL reconstruction on both domains.
• Introduce DefRec, a family of pretext tasks that deform a region of a point cloud and require reconstructing the original region using Chamfer distance as the SSL loss.
• Describe three region-deformation schemes for DefRec: volume-based, feature-based, and sampling-based deformations, each selecting regions to deform and reconstruct.
• Introduce Point Cloud Mixup (PCM), a Mixup-inspired procedure that creates a new labeled point cloud by sampling γn points from one cloud and (1−γ)n from another, with labels blended linearly.
• Define the overall loss as a weighted sum of the supervised loss on the source (or augmented with PCM) and the SSL reconstruction loss on deformed samples.
• Apply DefRec primarily on target data to bridge domain gaps, and show PCM improves baseline performance when combined with DefRec.

Experimental results

Research questions

• RQ1 How can SSL pretext tasks be designed for domain adaptation in 3D point clouds?
• RQ2 Do deformation-based reconstruction tasks (DefRec) bridge the source-target distribution gap more effectively than existing SSL tasks for point clouds?
• RQ3 Does a Mixup-based strategy for point clouds (PCM) complement DefRec in DA, and how does it affect performance on classification and segmentation tasks?
• RQ4 What is the impact of different DeRec deformation schemes (volume-based, feature-based, sampling-based) on DA performance?
• RQ5 Can DefRec be extended to segmentation tasks and how does it perform on PointSegDA?

Key findings

• DefRec and DefRec+PCM outperform strong baselines on six PointDA-10 adaptations, with DefRec+PCM achieving the best overall performance.
• DefRec+PCM improves accuracy by about 5 percentage points on average compared to the best baseline and by about 5.5 points versus PointDAN.
• Volume-based, feature-based, and sampling-based DefRec variants show varying strengths, with volume-based deformations often yielding strong results; mid-scale region deformations tend to be more effective.
• Applying DefRec to target data (and optionally PCM on source data) yields substantial gains across sim-to-real and cross-dataset adaptations.
• DefRec can be extended to segmentation; DefRec+PCM also improves segmentation mean IoU on PointSegDA over baselines.