[Paper Review] ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation

Fei Xia, Chengshu Li+4 more|arXiv (Cornell University)|Aug 18, 2020

Reinforcement Learning in RoboticsComputer Science58 references38 citations

TL;DR

ReLMoGen integrates a motion generator with reinforcement learning by lifting the action space to subgoals for motion planning, enabling efficient solving of long-horizon mobile manipulation tasks and demonstrating strong transferability across motion planners.

ABSTRACT

Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous control tasks. We propose to lift the action space to a higher level in the form of subgoals for a motion generator (a combination of motion planner and trajectory executor). We argue that, by lifting the action space and by leveraging sampling-based motion planners, we can efficiently use RL to solve complex, long-horizon tasks that could not be solved with existing RL methods in the original action space. We propose ReLMoGen -- a framework that combines a learned policy to predict subgoals and a motion generator to plan and execute the motion needed to reach these subgoals. To validate our method, we apply ReLMoGen to two types of tasks: 1) Interactive Navigation tasks, navigation problems where interactions with the environment are required to reach the destination, and 2) Mobile Manipulation tasks, manipulation tasks that require moving the robot base. These problems are challenging because they are usually long-horizon, hard to explore during training, and comprise alternating phases of navigation and interaction. Our method is benchmarked on a diverse set of seven robotics tasks in photo-realistic simulation environments. In all settings, ReLMoGen outperforms state-of-the-art Reinforcement Learning and Hierarchical Reinforcement Learning baselines. ReLMoGen also shows outstanding transferability between different motion generators at test time, indicating a great potential to transfer to real robots.

Motivation & Objective

• Motivate and address the exploration and long-horizon challenges in mobile manipulation tasks.
• Propose a framework that lifts actions to subgoals for a motion generator within an RL loop.
• Demonstrate improved performance and sample efficiency across navigation, interactive navigation, and mobile manipulation tasks.

Proposed method

• Introduce a lifted MDP where subgoals a' guide a motion generator (MG) to produce low-level actions.
• Two subgoal generation policy variants: continuous (SGP-R) and discrete (SGP-D), trained with SAC or DQN respectively.
• A motion generator combining a planner (RRT-Connect or PRM) and a trajectory controller to reach subgoals.
• Define lifted transition and reward functions: MG outputs a sequence of low-level actions; R' accumulates MG rewards over the sequence.
• Train the SGP to predict subgoals based on observations from RGB-D, LiDAR, and task info, enabling base and arm subgoals for navigation and interaction.
• Demonstrate transferability by swapping motion planners at test time without retraining.

Experimental results

Research questions

• RQ1 Can ReLMoGen solve a wide range of robotic tasks involving navigation and manipulation?
• RQ2 Does lifting the action space to subgoals improve exploration and sample efficiency in long-horizon mobile manipulation tasks?
• RQ3 Is the learned subgoal generation policy robust to changes in the motion planner at test time?
• RQ4 How do continuous and discrete subgoal parameterizations compare across tasks with varying manipulation demands?

Key findings

• ReLMoGen achieves higher task completion across seven tasks compared to state-of-the-art RL and HRL baselines.
• ReLMoGen exhibits faster convergence and greater sample efficiency, with training often 7x faster in wall-clock time due to fewer gradient updates.
• The approach yields interpretable subgoal maps, where high-value regions align with beneficial interactions (e.g., buttons, cabinet doors).
• ReLMoGen enables transfer to different motion planners at test time with minimal performance loss, showing strong robustness and practicality for real robots.
• SGP-D (discrete subgoal maps) outperforms in tasks requiring fine-grained manipulation, while SGP-R (continuous subgoal regression) excels in broader navigation and interaction scenarios.
• Across exploration analyses, ReLMoGen explores meaningful interactions and covers larger physical and latent state regions than a purely action-space RL baseline.