[Paper Review] Foundation Model based Open Vocabulary Task Planning and Executive System for General Purpose Service Robots

Yoshiki Obinata, Naoaki Kanazawa+5 more|arXiv (Cornell University)|Aug 7, 2023

Robotics and Automated SystemsEngineering3 citations

TL;DR

This paper proposes a foundation model-based open-vocabulary task planning and executive system for general-purpose service robots, integrating large language models (LLMs) and vision-language models (VLMs) with a state machine executor to enable robust, stable execution of spoken language commands in real-world environments. The system achieved first place in RoboCup@home Japan Open 2022 GPSR with 130 points, significantly outperforming other teams.

ABSTRACT

This paper describes a strategy for implementing a robotic system capable of performing General Purpose Service Robot (GPSR) tasks in robocup@home. The GPSR task is that a real robot hears a variety of commands in spoken language and executes a task in a daily life environment. To achieve the task, we integrate foundation models based inference system and a state machine task executable. The foundation models plan the task and detect objects with open vocabulary, and a state machine task executable manages each robot's actions. This system works stable, and we took first place in the RoboCup@home Japan Open 2022's GPSR with 130 points, more than 85 points ahead of the other teams.

Motivation & Objective

• To enable general-purpose service robots to understand and execute diverse spoken language commands in daily-life environments.
• To address the challenge of open-vocabulary understanding and action planning using foundation models.
• To ensure stable and deterministic robot behavior through a state machine-based executive system despite failures or ambiguities.
• To integrate local environmental knowledge and handle failures through human feedback and iterative refinement.
• To improve robustness in real-world execution by combining LLMs, VLMs, and executable state machines.

Proposed method

• The system uses a large language model (GPT-3) to parse spoken commands into a sequence of primitive actions.
• Vision-language models (OFA, Detic) enable open-vocabulary object detection and visual question answering.
• A state machine task executor manages action sequences, with deterministic transitions on success or failure of each primitive function.
• Primitive functions include move_to, grasp, follow, pass_to, speak, answer, and visual_question_answering.
• Local knowledge is injected into the LLM via natural language descriptions of the environment, such as object locations and spatial relationships.
• Failure handling includes retrying actions, asking humans for guidance, and using feedback to improve future execution.

Experimental results

Research questions

• RQ1 How can foundation models be effectively combined with a state machine executive to enable stable, deterministic robot behavior in open-vocabulary, real-world tasks?
• RQ2 Can LLMs and VLMs jointly enable robust open-vocabulary understanding of complex, natural language commands in daily-life environments?
• RQ3 How can local environmental knowledge—such as object locations and spatial layouts—be effectively integrated into foundation model reasoning?
• RQ4 What mechanisms can improve failure detection and recovery in robotic task execution when primitive functions fail unpredictably?
• RQ5 To what extent can a hybrid system of foundation models and state machines outperform traditional rule-based or learning-only approaches in real-world service robot competitions?

Key findings

• The proposed system achieved first place in the RoboCup@home Japan Open 2022 GPSR competition with a total score of 130 points.
• The system scored more than 85 points ahead of the second-place team, demonstrating superior performance in open-vocabulary task execution.
• The integration of LLMs for task planning and VLMs for visual understanding enabled robust open-vocabulary object detection and language grounding.
• The state machine executor ensured deterministic behavior, improving stability and reliability during task execution despite failures.
• The system effectively handled ambiguous or incomplete commands by prompting users for clarification when primitive functions failed.
• The researchers identified key challenges in detecting subtle failures (e.g., object drop during grasp) and emphasized the need for improved feedback mechanisms and data collection.