[Paper Review] DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback

Riku Arakawa, Sosuke Kobayashi+3 more|arXiv (Cornell University)|Oct 28, 2018

Reinforcement Learning in Robotics23 references48 citations

TL;DR

DQN-TAMER combines a Q-learning branch and a TAMER-style human feedback branch to learn faster under realistic human feedback conditions (binary, delayed, stochastic, unsustainable, and natural reaction). It outperforms baselines in Maze and Taxi and can leverage implicit facial-expression feedback in a real-world demo.

ABSTRACT

Exploration has been one of the greatest challenges in reinforcement learning (RL), which is a large obstacle in the application of RL to robotics. Even with state-of-the-art RL algorithms, building a well-learned agent often requires too many trials, mainly due to the difficulty of matching its actions with rewards in the distant future. A remedy for this is to train an agent with real-time feedback from a human observer who immediately gives rewards for some actions. This study tackles a series of challenges for introducing such a human-in-the-loop RL scheme. The first contribution of this work is our experiments with a precisely modeled human observer: binary, delay, stochasticity, unsustainability, and natural reaction. We also propose an RL method called DQN-TAMER, which efficiently uses both human feedback and distant rewards. We find that DQN-TAMER agents outperform their baselines in Maze and Taxi simulated environments. Furthermore, we demonstrate a real-world human-in-the-loop RL application where a camera automatically recognizes a user's facial expressions as feedback to the agent while the agent explores a maze.

Motivation & Objective

• Motivate reinforcement learning for robotics by addressing long-horizon credit assignment and slow reward propagation.
• Model realistic human-in-the-loop feedback including binary, delay, stochasticity, unsustainability, and natural reactions.
• Develop an algorithm that effectively fuses immediate human feedback with distant environmental rewards to speed up learning.
• Demonstrate robustness of the method in simulations and a real-world camera-based facial-expression feedback system.

Proposed method

• Introduce two value functions: Q for environment rewards and H for human feedback.
• Define a combined policy pi(s) = argmax_a [ alpha_q * Q_hat(s,a) + alpha_h * H_hat(s,a) ], where alpha_h decays to zero over time.
• Extend DQN with an auxiliary TAMER-like module to learn H through binary feedback using a loss L(H_hat) and episodic/local/global replay buffers.
• Allow delayed, stochastic, and potentially imperfect feedback, and show robustness to feedback stop (unsustainability) and natural reaction errors.
• Provide a real-time system where facial expressions are classified to generate implicit feedback and integrated into DQN-TAMER.

Experimental results

Research questions

• RQ1 How does DQN-TAMER perform when human feedback is binary, delayed, stochastic, and may stop before convergence?
• RQ2 Can the agent effectively combine immediate human feedback with distant environmental rewards to accelerate learning?
• RQ3 Is the method robust to recognition errors in implicit feedback (e.g., facial expression classifiers)?
• RQ4 Does the approach extend to real-world human-in-the-loop scenarios beyond simulated feedback?

Key findings

• DQN-TAMER learns faster and achieves better performance than DQN with reward shaping and Deep TAMER across Maze and Taxi tasks.
• The method is robust to delay, stochasticity, and feedback unsustainability, maintaining superior performance where baselines falter.
• Initial guidance from human feedback accelerates exploration, while environmental rewards dominate once decoupled by decaying alpha_h.
• In a real-world demo, a car robot using facial-expression feedback (happy/negative) can learn maze navigation despite classifier errors (~15% misclassification).
• The approach generalizes TAMER and DQN, behaving as DQN when alpha_h is zero and as Deep TAMER when alpha_q is zero.