[Paper Review] PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation

Shizhe Chen, Ricardo García+2 more|arXiv (Cornell University)|Sep 27, 2023

Multimodal Machine Learning Applications50 references4 citations

TL;DR

PolarNet proposes a 3D point cloud-based policy for language-guided robotic manipulation, leveraging multi-view point cloud fusion, efficient PointNext encoding, and multimodal transformers to predict 7-DoF actions. It achieves state-of-the-art performance on RLBench across single- and multi-task settings and attains 60% average success rate on real-world tasks, demonstrating robustness to viewpoint changes and data efficiency.

ABSTRACT

The ability for robots to comprehend and execute manipulation tasks based on natural language instructions is a long-term goal in robotics. The dominant approaches for language-guided manipulation use 2D image representations, which face difficulties in combining multi-view cameras and inferring precise 3D positions and relationships. To address these limitations, we propose a 3D point cloud based policy called PolarNet for language-guided manipulation. It leverages carefully designed point cloud inputs, efficient point cloud encoders, and multimodal transformers to learn 3D point cloud representations and integrate them with language instructions for action prediction. PolarNet is shown to be effective and data efficient in a variety of experiments conducted on the RLBench benchmark. It outperforms state-of-the-art 2D and 3D approaches in both single-task and multi-task learning. It also achieves promising results on a real robot.

Motivation & Objective

• To address the limitations of 2D image-based policies in language-guided robotic manipulation, such as poor 3D spatial reasoning and sensitivity to viewpoint changes.
• To explore the effectiveness of 3D point cloud representations for integrating language instructions with precise 3D action prediction in multi-task settings.
• To design an efficient, data-efficient, and robust policy that generalizes across multiple manipulation tasks using unified 3D point cloud inputs.
• To improve robustness to camera viewpoint perturbations by leveraging geometric consistency in 3D point clouds over 2D image-based methods.
• To enable real-world deployment of language-conditioned policies through fine-tuning on real-robot demonstrations.

Proposed method

• Constructs a unified 3D point cloud from multi-view RGB-D cameras using calibrated extrinsic parameters, incorporating color and depth features.
• Applies point cloud filtering to remove irrelevant points (e.g., background, robot parts) and improve representation quality.
• Employs PointNext as the backbone encoder to efficiently process sparse 3D point clouds with local and global context modeling.
• Uses a multimodal transformer to attend over both encoded point cloud features and tokenized language instructions for cross-modal fusion.
• Predicts 7-DoF actions (3D position, 3D rotation, gripper open/close) via a head network, executed by a motion planner.
• Trains end-to-end using imitation learning on the RLBench benchmark with supervised demonstration data, fine-tuning on real-robot data.

Experimental results

Research questions

• RQ1 Can 3D point cloud representations outperform 2D image-based methods in language-guided robotic manipulation, especially in multi-task and multi-variation settings?
• RQ2 How does the design of point cloud input—such as coordinate frame, feature composition, and view fusion—affect policy performance?
• RQ3 To what extent does a point cloud-based policy generalize across diverse manipulation tasks and handle viewpoint variations?
• RQ4 Can a 3D point cloud policy achieve high success rates on real-world robotic hardware with limited real-world data?
• RQ5 How does the model’s robustness to camera pose perturbations compare to 2D image-based baselines?

Key findings

• PolarNet achieves a 78.3% success rate on the multi-task, multi-variation RLBench setup (18 tasks, 249 variations), outperforming prior 2D and 3D methods.
• It improves by +11.1% in success rate over prior models in the multi-task setting, demonstrating strong generalization across tasks.
• PolarNet is 13 times faster to train than PerAct, requiring only 30 hours on 4 V100 GPUs versus 16 days on 8 V100s.
• On real-world tasks, PolarNet achieves a 60% average success rate across 7 tasks after fine-tuning on 20 human demonstrations per task.
• In viewpoint perturbation evaluation, PolarNet shows significantly lower performance drop than Hiveformer (2D-based), confirming robustness to camera placement changes.
• Failure cases are primarily due to object confusion (e.g., strawberry vs. apple) and imprecise grasping pose predictions, indicating limitations in semantic and spatial precision.