[Paper Review] Should artificial agents ask for help in human-robot collaborative problem-solving?

Adrien Bennetot, Vicky Charisi+1 more|arXiv (Cornell University)|May 25, 2020

Child and Animal Learning Development51 references5 citations

TL;DR

This paper investigates whether artificial agents should proactively ask for help in human-robot collaborative problem-solving by simulating a child-robot interaction study using a Q-learning agent. It finds that both children and the RL agent benefit equally from expert help—whether requested or provided canonically—demonstrating that help accelerates learning regardless of initiation method.

ABSTRACT

Transferring as fast as possible the functioning of our brain to artificial intelligence is an ambitious goal that would help advance the state of the art in AI and robotics. It is in this perspective that we propose to start from hypotheses derived from an empirical study in a human-robot interaction and to verify if they are validated in the same way for children as for a basic reinforcement learning algorithm. Thus, we check whether receiving help from an expert when solving a simple close-ended task (the Towers of Hanoï) allows to accelerate or not the learning of this task, depending on whether the intervention is canonical or requested by the player. Our experiences have allowed us to conclude that, whether requested or not, a Q-learning algorithm benefits in the same way from expert help as children do.

Motivation & Objective

• To investigate whether artificial agents should proactively ask for help in collaborative problem-solving, inspired by human developmental learning.
• To test whether hypotheses derived from empirical human-robot interaction (HRI) studies generalize to artificial agents using reinforcement learning.
• To compare the learning efficiency of a Q-learning agent when help is either canonically provided or requested by the agent, in a simplified task (Towers of Hanoi).
• To assess whether intrinsic motivation and uncertainty signaling in agents can be modeled to reflect human-like help-seeking behavior.
• To explore the role of expert intervention in accelerating learning, regardless of whether help is initiated by the agent or the environment.

Proposed method

• Simulates a human-robot interaction scenario using a Q-learning agent (LA1) and a help-seeking agent (LA2) in a simplified Towers of Hanoi task.
• Implements two configurations: canonical help (expert intervenes when the agent's best action is suboptimal) and requested help (agent asks when its best Q-value falls below a threshold).
• Uses a threshold-based mechanism to trigger help-seeking behavior in the LA2, based on the agent's confidence in its action selection.
• Employs a fixed reward structure and state representation to isolate the effect of help on learning speed and accuracy.
• Compares learning performance across episodes using metrics such as task completion time and success rate.
• Validates findings by contrasting agent behavior with observed developmental patterns in children, particularly in executive function and uncertainty monitoring.

Experimental results

Research questions

• RQ1 Does expert help accelerate learning in a Q-learning agent when help is requested versus provided canonically?
• RQ2 Do artificial agents benefit from help in the same way as children do in collaborative problem-solving tasks?
• RQ3 Can a simple threshold-based help-seeking mechanism in an RL agent replicate human-like help-seeking behavior observed in children?
• RQ4 How does the timing and frequency of help requests affect learning efficiency in artificial agents?
• RQ5 To what extent does the mode of help delivery (requested vs. canonical) influence the agent’s ability to learn and generalize strategies?

Key findings

• The Q-learning agent benefits equally from expert help whether the help is requested by the agent or provided canonically, mirroring findings in children.
• The LA2, which proactively asks for help when its confidence is low, achieves faster learning than the LA1 that never requests help, though it initially loses efficiency due to over-reliance on assistance.
• After an initial phase of dependency on help, the LA2 regains efficiency by exploring and learning independently, demonstrating adaptive learning behavior.
• The difference in performance between canonical and requested help configurations is minimal, suggesting that the mechanism of help delivery is less critical than the presence of help itself.
• The results support the hypothesis that help-seeking behavior, even in a simplified form, accelerates learning in artificial agents, similar to human children.
• The study indicates that future agents should incorporate more nuanced uncertainty modeling to better emulate human-like help-seeking, rather than relying on fixed thresholds.