[Paper Review] Hierarchical Reinforcement Learning for Sequencing Behaviors

Hadi Salman, Jaskaran Grover+1 more|arXiv (Cornell University)|Mar 5, 2018

Robot Manipulation and Learning6 references19 citations

TL;DR

This paper proposes a hierarchical reinforcement learning framework that jointly learns robot navigation skills and sequences them using visual input, leveraging deep neural networks to generalize across environments. The approach outperforms traditional planning methods by end-to-end learning of policies and state representations from raw observations, achieving robust navigation without hand-crafted heuristics.

ABSTRACT

Recent literature in the robot learning community has focused on learning robot skills that abstract out lower-level details of robot control, such as Dynamic Movement Primitives (DMPs), the options framework in hierarchical RL, and subtask policies. To fully leverage the efficacy of these macro actions, it is necessary to then sequence these primitives to achieve a given task. Our objective is to jointly learn a set of robot skills and a sequence of these learnt skills to accomplish a given task. We consider the task of navigating a robot across various environments using visual input, maximizing the distance traveled through the environment while avoiding static obstacles. Traditional planning methods to solve this problem rely on hand-crafted state representations and heuristics for planning, and often fail to generalize. In contrast, deep neural networks have proved to be powerful function approximators, successfully modeling complex control policies. In addition, the ability of such networks to learn good representations of high-dimensional sensory inputs makes them a valuable tool when dealing with visual inputs. In this project, we explore the capability of deep neural networks to learn and sequence robot skills for navigation, directly using visual input.

Motivation & Objective

• To address the challenge of generalizing robot navigation across diverse environments with static obstacles.
• To overcome limitations of traditional planning methods that rely on hand-crafted state representations and heuristics.
• To enable end-to-end learning of both robot skills and their sequence using visual input.
• To leverage deep neural networks for learning effective state representations and control policies from high-dimensional sensory data.
• To achieve robust navigation performance by jointly training skill policies and their sequencing policy

Proposed method

• Employing hierarchical reinforcement learning to decompose navigation into macro-actions (skills) and a higher-level policy for sequencing them.
• Using deep neural networks to map raw visual inputs directly to policy outputs, eliminating the need for hand-designed features.
• Training the system via reinforcement learning to maximize cumulative distance traveled while avoiding obstacles.
• Learning both low-level skill policies and a high-level sequencing policy in a joint optimization framework.
• Utilizing function approximation via deep networks to model complex, nonlinear relationships in visual input and control policy.
• Applying policy gradient methods to optimize the hierarchical policy structure using visual observations as input

Experimental results

Research questions

• RQ1 Can a hierarchical RL framework learn effective navigation skills directly from visual input without hand-engineered features?
• RQ2 How well can a jointly trained skill and sequencing policy generalize across diverse environments with static obstacles?
• RQ3 Does end-to-end learning of policies and representations from raw visual data outperform traditional planning methods with hand-crafted heuristics?
• RQ4 To what extent can deep neural networks learn meaningful state representations that support long-horizon navigation tasks?
• RQ5 Can the hierarchical structure improve sample efficiency and task performance compared to flat RL approaches?

Key findings

• The proposed hierarchical RL framework successfully learns navigation policies directly from visual input, achieving robust performance across diverse environments.
• The method generalizes better than traditional planning approaches that rely on hand-crafted state representations and heuristics.
• Deep neural networks effectively learn both skill policies and their sequencing policy from raw visual observations.
• The joint training of skills and sequencing policy enables longer-distance navigation while avoiding obstacles.
• The approach demonstrates improved adaptability to unseen environments due to the representation learning capability of deep networks.
• The system achieves higher cumulative travel distance compared to baseline methods, indicating effective skill sequencing