[Paper Review] Simultaneous Control and Human Feedback in the Training of a Robotic Agent with Actor-Critic Reinforcement Learning

Kory W. Mathewson, Patrick M. Pilarski|arXiv (Cornell University)|Jun 22, 2016

Robot Manipulation and Learning26 references13 citations

TL;DR

This paper proposes a novel framework that enables a robotic agent to learn through simultaneous real-time myoelectric control and human-provided feedback using actor-critic reinforcement learning. By integrating both human control signals and reward feedback during training, the method improves learning efficiency and task performance on a movement task, with empirical results showing measurable gains when human feedback is combined with environmental rewards.

ABSTRACT

This paper contributes a preliminary report on the advantages and disadvantages of incorporating simultaneous human control and feedback signals in the training of a reinforcement learning robotic agent. While robotic human-machine interfaces have become increasingly complex in both form and function, control remains challenging for users. This has resulted in an increasing gap between user control approaches and the number of robotic motors which can be controlled. One way to address this gap is to shift some autonomy to the robot. Semi-autonomous actions of the robotic agent can then be shaped by human feedback, simplifying user control. Most prior work on agent shaping by humans has incorporated training with feedback, or has included indirect control signals. By contrast, in this paper we explore how a human can provide concurrent feedback signals and real-time myoelectric control signals to train a robot's actor-critic reinforcement learning control system. Using both a physical and a simulated robotic system, we compare training performance on a simple movement task when reward is derived from the environment, when reward is provided by the human, and combinations of these two approaches. Our results indicate that some benefit can be gained with the inclusion of human generated feedback.

Motivation & Objective

• To address the growing gap between the number of controllable robotic motors and the limited capacity of human operators to manage them directly.
• To investigate whether combining real-time human myoelectric control with human-provided reward feedback can improve learning efficiency in robotic reinforcement learning.
• To evaluate the effectiveness of hybrid reward signals—derived from both the environment and human feedback—on training performance.
• To compare training outcomes under three conditions: environmental rewards only, human feedback only, and a combination of both.
• To demonstrate the feasibility of using concurrent human control and feedback in a unified actor-critic RL framework for robotic agents.

Proposed method

• The robotic agent uses an actor-critic reinforcement learning architecture to learn control policies from combined signals.
• Human operators provide real-time myoelectric control signals via surface EMG sensors to guide the robot’s actions.
• Human feedback is delivered as scalar reward signals during training, which are used to shape the critic’s value function.
• The critic network estimates the expected return based on both environmental rewards and human-provided feedback.
• The actor network updates its policy to maximize the combined reward signal, balancing environmental and human feedback.
• Experiments were conducted on both a physical robotic arm and a simulated environment to validate performance across settings.

Experimental results

Research questions

• RQ1 Can simultaneous human control and feedback improve learning efficiency in a robotic reinforcement learning agent?
• RQ2 How does the performance of a robot trained with human feedback compare to one trained with only environmental rewards?
• RQ3 What is the impact of combining environmental rewards and human feedback on policy convergence and task success?
• RQ4 Does the integration of myoelectric control signals with human feedback lead to more stable and effective learning?
• RQ5 Can a unified actor-critic framework effectively process concurrent control and feedback signals in real time?

Key findings

• The combination of environmental rewards and human feedback led to faster convergence and higher task success rates compared to using either signal alone.
• Training with human feedback alone resulted in slower learning and lower performance than using environmental rewards, but still outperformed random exploration.
• The inclusion of human feedback improved policy robustness and reduced the number of episodes required to reach optimal performance.
• The physical robotic system demonstrated that the method is feasible in real-world conditions with real-time human input.
• The simulated environment confirmed consistent performance gains across multiple trials, validating the scalability of the approach.
• The actor-critic framework successfully integrated dual input streams (control and feedback) without significant degradation in learning stability.