[Paper Review] LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robots

Dongge Han, Trevor McInroe+4 more|arXiv (Cornell University)|Apr 22, 2024

Robot Manipulation and Learning4 citations

TL;DR

LLM-Personalize introduces a reinforcement-based self-training pipeline that personalizes LLM planners for household robots by combining imitation learning and iterative self-training to align with user preferences. It achieves over a 30% increase in success rate on the Housekeep benchmark by improving plan executability and preference alignment in multi-room, partially observable environments.

ABSTRACT

Large language models (LLMs) have shown significant potential for robotics applications, particularly task planning, by harnessing their language comprehension and text generation capabilities. However, in applications such as household robotics, a critical gap remains in the personalization of these models to individual user preferences. We introduce LLM-Personalize, a novel framework with an optimization pipeline designed to personalize LLM planners for household robotics. Our LLM-Personalize framework features an LLM planner that performs iterative planning in multi-room, partially-observable household scenarios, making use of a scene graph constructed with local observations. The generated plan consists of a sequence of high-level actions which are subsequently executed by a controller. Central to our approach is the optimization pipeline, which combines imitation learning and iterative self-training to personalize the LLM planner. In particular, the imitation learning phase performs initial LLM alignment from demonstrations, and bootstraps the model to facilitate effective iterative self-training, which further explores and aligns the model to user preferences. We evaluate LLM-Personalize on Housekeep, a challenging simulated real-world 3D benchmark for household rearrangements, and show that LLM-Personalize achieves more than a 30 percent increase in success rate over existing LLM planners, showcasing significantly improved alignment with human preferences. Project page: https://gdg94.github.io/projectllmpersonalize/.

Motivation & Objective

• To address the gap in personalizing LLM planners for household robotics to align with individual user preferences.
• To improve long-horizon planning in multi-room, partially observable environments using LLMs.
• To develop a scalable optimization pipeline that enables effective personalization without requiring extensive human-annotated preference data.
• To evaluate the framework on a challenging 3D simulated benchmark, Housekeep, to demonstrate real-world applicability.
• To enable cross-domain transfer of personalized planning behavior to unseen household layouts.

Proposed method

• The framework uses a three-component agent architecture: a context generator that builds and updates a scene graph from local observations, an LLM planner that generates high-level action sequences based on the scene graph, and a low-level controller that executes the actions.
• An initial imitation learning phase aligns the LLM planner with human demonstration data to bootstrap context understanding and preference interpretation.
• An iterative self-training phase follows, where the model generates interaction data, evaluates outcomes based on user preference feedback, and updates its policy to favor high-reward (preferred) placements.
• The self-training process uses a reward signal derived from human preference data to optimize for correct object-receptacle placements, balancing exploration and exploitation.
• The optimization pipeline is applied in a bootstrapped manner, starting from imitation learning to improve data quality before self-training begins.
• The framework is evaluated on the Housekeep benchmark, which features diverse, long-horizon, partially observable household rearrangement tasks with real human preference annotations.

Figure 1 : Illustration of LLM-Personalize. Agent architecture: The Context Generator constructs and updates a scene graph from local observations. The LLM Planner uses the graph to produce a plan as a sequence of high-level actions, and iteratively re-plans when the previous plan has been executed.

Experimental results

Research questions

• RQ1 Can an LLM planner be effectively personalized to individual user preferences in household robotics without requiring explicit preference modeling?
• RQ2 How does combining imitation learning with iterative self-training improve plan quality and preference alignment in multi-room, partially observable environments?
• RQ3 To what extent can the LLM-Personalize framework generalize to unseen household layouts and object-receptacle combinations?
• RQ4 How does the balance between exploration and exploitation evolve during the self-training process, and what impact does it have on final task performance?
• RQ5 What is the contribution of each optimization phase—imitation learning versus self-training—to the overall success rate in preference-aligned planning?

Key findings

• LLM-Personalize achieves more than a 30% increase in success rate over existing LLM-based planners on the Housekeep benchmark, demonstrating strong preference alignment.
• The imitation learning phase significantly improves plan executability, increasing the percentage of successfully executed high-level actions and enabling high-quality self-training data.
• During self-training, the agent increases exploration, with unique placements rising from 2.32 to 9.48 in Scene 3, before shifting toward exploitation in later stages.
• The model generalizes well to unseen object combinations and layouts, with test performance improving when trained on diverse scenes, indicating effective cross-domain transfer.
• Overfitting was observed in Scene 3, where test performance dropped despite high training performance, suggesting a need for regularization in complex scenarios.
• The framework successfully learns and commits to correct object placements based on user preferences, avoiding incorrect placements through iterative optimization.

Figure 2 : The Context generator model maintains and updates the graph of the household state including rooms, receptacles and objects, derived from the robot’s local observations at each timestep. The information is provided as a prompt for the LLM planner. The top-down view of the scene is for ill