[Paper Review] PaLM-E: An Embodied Multimodal Language Model

Danny Driess, Fei Xia+20 more|arXiv (Cornell University)|Mar 6, 2023

Multimodal Machine Learning ApplicationsComputer Science349 citations

TL;DR

PaLM-E integrates continuous observations (images, state estimates, etc.) into a single decoder-only large language model to perform embodied reasoning, visual-language tasks, and traditional language tasks, showing transfer across robotics and vision-language domains and achieving state-of-the-art OK-VQA performance without task-specific fine-tuning.

ABSTRACT

Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.

Motivation & Objective

• Ground the language model in real-world continuous sensor modalities to enable embodied reasoning.
• Create a single multimodal model that handles robotic planning, visual question answering, and captioning.
• Show transfer from diverse vision-language data to embodied tasks and across robot embodiments.
• Demonstrate that scaling the model and multi-task training improve data efficiency and generalization.

Proposed method

• Inject continuous observations (images, state estimates) as multimodal tokens into a pre-trained decoder-only LLM to form multimodal sentences.
• Train encoders for each modality end-to-end with the LLM to generate text that can be interpreted as plans or answers.
• Use object-centric representations (OSRT, ground-truth object masks) to tokenize objects into multiple embeddings and enable object references in text.
• Evaluate different input representations (state vectors, ViT variants, OSRT) and training regimes (freezing vs fine-tuning the LLM, multi-task co-training).
• Connect PaLM-E outputs to low-level policies in a control loop, treating PaLM-E as a high-level planner that sequences low-level skills.

Experimental results

Research questions

• RQ1 Can a single multimodal language model grounded with continuous sensor modalities perform both embodied reasoning and standard vision-language tasks?
• RQ2 Does multi-task, cross-domain training improve data efficiency and enable transfer across robot embodiments and tasks?
• RQ3 How do different modality encoders (state vectors, ViT variants, OSRT) affect performance in embodied planning and VQA?
• RQ4 What is the impact of freezing vs finetuning the LLM and of object-centric representations on generalization and data efficiency?
• RQ5 Can scaling PaLM-E to large parameter counts retain generalist capabilities while enabling embodied planning and zero-shot multimodal reasoning?

Key findings

• PaLM-E achieves strong performance across embodied planning, VQA, and captioning, and enables zero-shot multimodal chain-of-thought reasoning.
• Co-training on a mixture of robotics and general vision-language data improves performance compared to in-domain data alone, even with limited embodied data.
• OSRT-based 3D-aware object representations provide the best planning performance in the TAMP environment among tested encoders.
• The 562B-parameter PaLM-E model attains state-of-the-art OK-VQA results without task-specific fine-tuning and demonstrates transfer across multiple tasks and embodiments.
• Freezing the LLM and training only the encoders is viable in some settings, suggesting a soft-prompting effect through modality-grounded encoders.