[Paper Review] 3D-VLA: A 3D Vision-Language-Action Generative World Model

Haoyu Zhen, Xiaowen Qiu+6 more|arXiv (Cornell University)|Mar 14, 2024

Robotics and Automated Systems14 citations

TL;DR

3D-VLA introduces a 3D vision-language-action embodied foundation model that reasons in 3D, generates goal images/point clouds, and predicts actions for manipulation by integrating diffusion-based goal generation with a 3D-LMM backbone and interaction tokens.

ABSTRACT

Recent vision-language-action (VLA) models rely on 2D inputs, lacking integration with the broader realm of the 3D physical world. Furthermore, they perform action prediction by learning a direct mapping from perception to action, neglecting the vast dynamics of the world and the relations between actions and dynamics. In contrast, human beings are endowed with world models that depict imagination about future scenarios to plan actions accordingly. To this end, we propose 3D-VLA by introducing a new family of embodied foundation models that seamlessly link 3D perception, reasoning, and action through a generative world model. Specifically, 3D-VLA is built on top of a 3D-based large language model (LLM), and a set of interaction tokens is introduced to engage with the embodied environment. Furthermore, to inject generation abilities into the model, we train a series of embodied diffusion models and align them into the LLM for predicting the goal images and point clouds. To train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by extracting vast 3D-related information from existing robotics datasets. Our experiments on held-in datasets demonstrate that 3D-VLA significantly improves the reasoning, multimodal generation, and planning capabilities in embodied environments, showcasing its potential in real-world applications.

Motivation & Objective

• Motivate the need for 3D embodied world models that integrate perception, reasoning, and action beyond 2D inputs.
• Develop a scalable 3D embodied foundation model that can reason, imagine multimodal goals, and plan actions in 3D environments.
• Create a large-scale 3D embodied instruction tuning dataset to train such models.
• Enable interaction with 3D environments via dedicated tokens and diffusion-based goal generation aligned to an LLM.

Proposed method

• Build a 3D-LLM backbone (3D-VLA) with dedicated interaction tokens for objects, locations, scenes, and 7-DoF robot actions.
• Inject goal generation by pretraining RGB-D-to-RGB-D and point-to-point diffusion models and align them to the LLM via a transformer projector.
• Curate a 2M 3D-language-action dataset by extracting 3D annotations (depth, point clouds, 3D boxes, actions) from robotics and human-object datasets.
• Use LoRA to fine-tune diffusion models while training only the new token embeddings, the output layer, and the projector.
• Bridge language and diffusion models with added tokens (<image>, <pcd>, etc.) and a projector to map DM features to LLM embeddings.
• Train a pipeline to derive 3D goal outputs (images, depths, point clouds) conditioned on instructions to guide embodied planning.

Experimental results

Research questions

• RQ1 Can a 3D-oriented world model outperform 2D baselines on embodied reasoning, localization, and planning?
• RQ2 To what extent can goal-generation diffusion models aligned to an LLM improve multimodal outputs (images, depths, point clouds) for embodied tasks?
• RQ3 Does a 3D embodied instruction tuning dataset enable robust 3D localization, task captioning, and action prediction in robotics settings?
• RQ4 How do interaction tokens and 3D representations affect action planning and generalization to unseen tasks?
• RQ5 What is the comparative performance of 3D-VLA versus 2D vision-language models on embodied tasks?

Key findings

• 3D-VLA outperforms 2D baselines on reasoning and localization tasks across embodied datasets.
• 3D-VLA achieves superior RGB and point-cloud goal generation compared to non-3D or non-aligned baselines, aided by intermediate predicted bounding boxes.
• The model demonstrates strong action planning on RLBench and CALVIN benchmarks, with notable improvements over selected baselines in multiple tasks.
• Localization results show 3D-VLA surpasses Kosmos-2 and CoVLM baselines when using 3D annotations.
• The dataset and 3D tokens enable robust interaction with dynamic scenes and improved manipulation predictions.
• A dedicated 3D embodied instruction tuning dataset is effective in enabling 3D reasoning, goal imagination, and action execution.