[Paper Review] Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems

Vinicius G. Goecks|arXiv (Cornell University)|Aug 30, 2020

Reinforcement Learning in Robotics169 references4 citations

TL;DR

This paper proposes Cycle-of-Learning, a human-in-the-loop framework that integrates human demonstrations, interventions, and evaluations into actor-critic reinforcement learning to improve sample efficiency and real-time adaptability in robotics. By iteratively cycling human feedback into the learning loop, the method accelerates convergence and achieves 1.84× higher task completion per human sample compared to baselines, surpassing behavior cloning and DAPG on UAS and LunarLander tasks.

ABSTRACT

Recent successes combine reinforcement learning algorithms and deep neural networks, despite reinforcement learning not being widely applied to robotics and real world scenarios. This can be attributed to the fact that current state-of-the-art, end-to-end reinforcement learning approaches still require thousands or millions of data samples to converge to a satisfactory policy and are subject to catastrophic failures during training. Conversely, in real world scenarios and after just a few data samples, humans are able to either provide demonstrations of the task, intervene to prevent catastrophic actions, or simply evaluate if the policy is performing correctly. This research investigates how to integrate these human interaction modalities to the reinforcement learning loop, increasing sample efficiency and enabling real-time reinforcement learning in robotics and real world scenarios. This novel theoretical foundation is called Cycle-of-Learning, a reference to how different human interaction modalities, namely, task demonstration, intervention, and evaluation, are cycled and combined to reinforcement learning algorithms. Results presented in this work show that the reward signal that is learned based upon human interaction accelerates the rate of learning of reinforcement learning algorithms and that learning from a combination of human demonstrations and interventions is faster and more sample efficient when compared to traditional supervised learning algorithms. Finally, Cycle-of-Learning develops an effective transition between policies learned using human demonstrations and interventions to reinforcement learning. The theoretical foundation developed by this research opens new research paths to human-agent teaming scenarios where autonomous agents are able to learn from human teammates and adapt to mission performance metrics in real-time and in real world scenarios.

Motivation & Objective

• To address the poor sample efficiency and risk of catastrophic failure in end-to-end reinforcement learning for real-world robotics.
• To integrate human interaction modalities—demonstrations, interventions, and evaluations—directly into the reinforcement learning loop.
• To enable real-time, data-efficient policy learning in high-fidelity simulated and real-world environments.
• To develop a theoretical framework that unifies human feedback with function approximation and reward shaping in actor-critic architectures.
• To validate the approach on continuous control tasks and unmanned air systems, demonstrating superior performance over standard RL and imitation learning.

Proposed method

• The Cycle-of-Learning framework integrates human demonstrations, interventions, and evaluations into the actor-critic architecture using function approximation for value and action functions.
• Human feedback is used to shape the reward signal via inverse reinforcement learning and generative modeling, improving policy learning efficiency.
• Experience replay incorporates both expert demonstrations and agent-generated experiences, with dynamic weighting based on loss components.
• The method employs a closed-loop interaction cycle where human inputs are continuously fed back into the learning process to refine policies iteratively.
• A pre-training phase using behavior cloning initializes the policy, followed by fine-tuning via reinforcement learning with human-in-the-loop corrections.
• Neural network architectures are optimized using techniques like Neural Architecture Search and compression to enable real-time deployment on hardware.

Experimental results

Research questions

• RQ1 How can human demonstrations, interventions, and evaluations be systematically integrated into the reinforcement learning loop to improve sample efficiency?
• RQ2 Can combining human feedback across multiple modalities lead to faster convergence and better performance than isolated feedback types?
• RQ3 Does integrating human prior knowledge into all components of the RL pipeline—beyond just reward shaping—yield measurable improvements?
• RQ4 How does the Cycle-of-Learning framework compare to traditional behavior cloning and end-to-end RL in terms of data efficiency and final policy performance?
• RQ5 To what extent can human-in-the-loop learning enable real-time adaptation and deployment in high-fidelity robotic environments?

Key findings

• The Cycle-of-Learning method achieved 636% better performance than behavior cloning on LunarLanderContinuous-v2, with 104% improvement over DAPG and 71% over DDPG.
• In the Microsoft AirSim UAS environment, the method showed 161% higher initial performance than DAPG and 297% higher than behavior cloning.
• Learning from a combination of demonstrations and interventions increased task completion by 12.8% (±3.6% std. error) while using 32.1% (±3.2% std. error) fewer human samples.
• The resulting policy had a task completion rate per human sample 1.84 times higher than baselines, demonstrating superior data efficiency.
• Component analysis confirmed that pre-training, loss weighting, and hybrid experience replay were critical to performance gains, not merely sequential application of imitation and RL.
• The agent learned to perform UAS landing tasks and surpassed human mean performance after only 100 learning iterations using inverse reinforcement learning and generative models.