[Paper Review] Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning

Haoyi Zhu, Yating Wang+4 more|arXiv (Cornell University)|Feb 4, 2024

Robotics and Sensor-Based Localization4 citations

TL;DR

This paper investigates the impact of different observation modalities—RGB, RGB-D, and 3D point clouds—on robot learning in contact-rich manipulation tasks. Using a standardized Action Chunking Transformer policy across 17 tasks in ManiSkill2 and RLBench, it demonstrates that point cloud observations consistently outperform RGB and RGB-D, achieving the highest success rates and superior zero-shot generalization across viewpoint, lighting, noise, and background variations.

ABSTRACT

In robot learning, the observation space is crucial due to the distinct characteristics of different modalities, which can potentially become a bottleneck alongside policy design. In this study, we explore the influence of various observation spaces on robot learning, focusing on three predominant modalities: RGB, RGB-D, and point cloud. We introduce OBSBench, a benchmark comprising two simulators and 125 tasks, along with standardized pipelines for various encoders and policy baselines. Extensive experiments on diverse contact-rich manipulation tasks reveal a notable trend: point cloud-based methods, even those with the simplest designs, frequently outperform their RGB and RGB-D counterparts. This trend persists in both scenarios: training from scratch and utilizing pre-training. Furthermore, our findings demonstrate that point cloud observations often yield better policy performance and significantly stronger generalization capabilities across various geometric and visual conditions. These outcomes suggest that the 3D point cloud is a valuable observation modality for intricate robotic tasks. We also suggest that incorporating both appearance and coordinate information can enhance the performance of point cloud methods. We hope our work provides valuable insights and guidance for designing more generalizable and robust robotic models. Codes are available at https://github.com/HaoyiZhu/PointCloudMatters.

Motivation & Objective

• To evaluate the relative effectiveness of RGB, RGB-D, and 3D point cloud observation modalities in robot learning.
• To investigate whether point clouds provide better zero-shot generalization under visual and geometric perturbations such as viewpoint changes, lighting, noise, and background variations.
• To assess performance gains from pretraining using state-of-the-art pretrained visual representations (PVRs) across different modalities.
• To establish a fair, unified comparison framework without domain-specific or complex architectural tricks.
• To provide empirical evidence that 3D spatial information from point clouds is more robust and effective than 2D appearance-based observations for complex manipulation tasks.

Proposed method

• Employs an Action Chunking Transformer (ACT) policy network as the controller across all observation modalities to ensure consistent policy architecture.
• Uses simple, widely adopted encoders for each modality: ResNet50, ViT-B, MultiViT-B, SpUNet34, and PVRs like R3M, VC-1, MultiMAE, and PonderV2.
• Evaluates performance on 17 contact-rich tasks in two benchmarks: ManiSkill2 and RLBench, using two simulators for robustness.
• Compares models trained from scratch and with pretraining to assess both sample efficiency and generalization.
• Measures zero-shot generalization across camera viewpoints, lighting intensity, rendering noise levels, and background colors.
• Uses mean success rate and mean rank as primary metrics across all evaluations.

Figure 1 : Point cloud has better zero-shot generalization ability on camera view and visual changes. First row: Zero-shot camera view generalization of different observations with training-from-scratch encoders ( left) and with SOTA PVRs ( right ) . Second row: Zero-shot generalization on different

Experimental results

Research questions

• RQ1 Does the 3D point cloud observation modality lead to higher performance than RGB and RGB-D in complex, contact-rich robotic manipulation tasks?
• RQ2 How does the zero-shot generalization capability of point cloud-based policies compare to RGB and RGB-D across changes in camera viewpoint, lighting, and background appearance?
• RQ3 To what extent do pretrained visual representations (PVRs) improve performance and generalization in different observation modalities?
• RQ4 Is the performance advantage of point clouds consistent across training-from-scratch and pretraining settings?
• RQ5 How does sample efficiency vary across different observation modalities when training policies on limited data?

Key findings

• Point cloud-based methods achieve the highest mean success rate (29%) and mean rank (1.22) across 17 tasks, outperforming RGB and RGB-D in both training-from-scratch and pretraining settings.
• Point cloud policies generalize effectively to unseen camera viewpoints, achieving a 29% success rate in zero-shot evaluation, compared to 18% for RGB and 23% for RGB-D.
• Under varying lighting conditions, point cloud models achieve up to 25.3% success rate in zero-shot generalization (e.g., 3.00 intensity), significantly outperforming RGB (0.00%) and RGB-D (0.00%) under extreme lighting.
• In noise-robustness tests, point cloud models maintain 21.0% success rate at high noise levels (64), while RGB and RGB-D models drop to 0.00% or near 0.00%.
• For background color changes, point cloud models achieve 36.8% success rate (R1.0, G1.0), far exceeding RGB (0.00%) and RGB-D (0.00%) under the same conditions.
• With only 10% of the training data, point cloud models maintain a mean success rate of 1.3% (PonderV2), while RGB and RGB-D models drop to 0.00% or near 0.00%, indicating superior sample efficiency.

Figure 2 : Overview of this work. We examine the impact of various observation spaces, specifically RGB, RGB-D, and point clouds, on robot learning. For this evaluation, we select prevalent and state-of-the-art methods, along with pretrained representations, and use them to train a action chunking t