[Paper Review] Leveraging Human Guidance for Deep Reinforcement Learning Tasks

Ruohan Zhang, Faraz Torabi+3 more|arXiv (Cornell University)|Sep 21, 2019

Reinforcement Learning in Robotics56 references4 citations

TL;DR

This paper surveys five deep reinforcement learning frameworks that leverage non-traditional human guidance—such as evaluative feedback, preferences, high-level goals, attention maps, and state sequences—instead of full action demonstrations. These methods reduce human effort, improve sample efficiency, and enable faster learning in complex tasks like Atari games and robotic control.

ABSTRACT

Reinforcement learning agents can learn to solve sequential decision tasks by interacting with the environment. Human knowledge of how to solve these tasks can be incorporated using imitation learning, where the agent learns to imitate human demonstrated decisions. However, human guidance is not limited to the demonstrations. Other types of guidance could be more suitable for certain tasks and require less human effort. This survey provides a high-level overview of five recent learning frameworks that primarily rely on human guidance other than conventional, step-by-step action demonstrations. We review the motivation, assumption, and implementation of each framework. We then discuss possible future research directions.

Motivation & Objective

• Address the high sample complexity of deep reinforcement learning by incorporating human guidance beyond full action demonstrations.
• Reduce reliance on costly, high-quality behavioral cloning data by using alternative human feedback signals.
• Explore how diverse forms of human guidance—such as feedback, preferences, and attention—can accelerate policy learning in complex environments.
• Investigate the trade-offs between information richness and human effort in human-in-the-loop RL frameworks.
• Promote reproducibility and benchmarking by advocating for shared datasets of human guidance in RL research.

Proposed method

• Use human evaluative feedback to provide scalar rewards or binary comparisons on agent behavior, enabling reward shaping without action-level demonstrations.
• Implement learning from human preference by comparing two agent trajectories and using pairwise comparisons to optimize policy preferences.
• Introduce hierarchical imitation learning where humans provide high-level goals, and low-level policies are trained to achieve them.
• Leverage human attention maps to guide the agent’s focus on relevant environmental features, improving perception and policy learning.
• Train agents using state sequences without actions, enabling imitation from observation by modeling state transition dynamics.
• Combine multiple guidance types in a unified lifelong learning framework to support adaptive, multi-stage learning.

Experimental results

Research questions

• RQ1 How can evaluative feedback be effectively used to shape rewards in deep RL without full action demonstrations?
• RQ2 To what extent do human preferences over behavior trajectories improve policy learning compared to traditional imitation learning?
• RQ3 Can high-level goals provided by humans significantly reduce the sample complexity of learning in hierarchical control tasks?
• RQ4 How does human attention guidance improve the agent’s ability to focus on relevant environmental features?
• RQ5 What is the impact of using state sequences without actions on learning performance in imitation-from-observation settings?

Key findings

• Human evaluative feedback and preference-based learning reduce the need for large-scale action demonstration data while still enabling effective policy optimization.
• High-level goal supervision in hierarchical frameworks improves sample efficiency and enables learning of long-horizon tasks in simulated robotic control.
• Attention-guided learning enhances agent performance by aligning the agent’s focus with human-relevant features, improving generalization.
• Imitation from observation using state sequences alone achieves competitive performance in environments where action data is unavailable or costly.
• Preference-based methods show strong performance in complex environments like Atari games, where human feedback is more scalable than expert demonstrations.
• The integration of multiple guidance types into a unified lifelong learning framework enables adaptive, multi-modal learning that outperforms single-guidance approaches.