[Paper Review] Augmenting Knowledge through Statistical, Goal-oriented Human-Robot Dialog

Saeid Amiri, Sujay Bajracharya+3 more|arXiv (Cornell University)|Jul 8, 2019

Speech and dialogue systems31 references4 citations

TL;DR

This paper presents a dual-track dialog manager that enables robots to simultaneously identify human service requests and augment their knowledge base (KB) with new entities and referring expressions through real-time interaction. By using a semantic parser and a probabilistic dialog manager, the system learns from dialog experiences to improve language understanding and KB completeness, outperforming static KB baselines in both efficiency and success rate during human-robot interactions.

ABSTRACT

Some robots can interact with humans using natural language, and identify service requests through human-robot dialog. However, few robots are able to improve their language capabilities from this experience. In this paper, we develop a dialog agent for robots that is able to interpret user commands using a semantic parser, while asking clarification questions using a probabilistic dialog manager. This dialog agent is able to augment its knowledge base and improve its language capabilities by learning from dialog experiences, e.g., adding new entities and learning new ways of referring to existing entities. We have extensively evaluated our dialog system in simulation as well as with human participants through MTurk and real-robot platforms. We demonstrate that our dialog agent performs better in efficiency and accuracy in comparison to baseline learning agents. Demo video can be found at https://youtu.be/DFB3jbHBqYE

Motivation & Objective

• To enable robots to learn new entities and referring expressions from human-robot dialog, extending beyond pre-defined knowledge bases.
• To design a dialog system that dynamically decides when and how to augment its KB based on dialog confidence and information gain.
• To improve the efficiency and accuracy of service request identification in multi-turn, natural language interactions.
• To evaluate the system in both simulation and real-world settings with human participants, including MTurk and real-robot trials.

Proposed method

• The system uses a hybrid semantic parser to interpret user utterances and extract intent, entities, and attributes.
• A dual-track Partially Observable Markov Decision Process (POMDP) controller manages both dialog flow and KB augmentation decisions in parallel.
• The dialog manager uses confidence tracking to determine when to ask clarification questions and when to initiate KB updates.
• The KB augmentation strategy adds new entities (e.g., people, items) only when the agent detects ambiguity or uncertainty, ensuring semantically meaningful additions.
• The system learns new referring expressions (e.g., 'me', 'Nate') by observing user input and updating the KB with new names and their referents.
• The agent is implemented on a Segway-based mobile robot using ROS and evaluated via simulation, MTurk, and real-world human trials.

Experimental results

Research questions

• RQ1 Can a robot dynamically augment its knowledge base during natural language dialog without relying on pre-defined entities?
• RQ2 How does on-the-fly KB augmentation affect dialog success rate and efficiency in human-robot interaction?
• RQ3 Does the dual-track dialog manager improve user experience compared to static KB baselines?
• RQ4 To what extent can the system learn new referring expressions (e.g., 'me', 'Nate') from user input?
• RQ5 How does the system’s ability to learn from dialog experience compare to fixed-interval KB update strategies?

Key findings

• Our dialog agent achieved significantly higher success rates and used fewer dialog turns than a baseline with a static KB, especially in real-robot trials.
• At the 0.1 confidence level, participants reported significantly less frustration (p=0.03) and greater willingness to use the robot in the future (p=0.02) when using our agent.
• The system successfully learned new entities such as 'Nate' and new referring expressions like 'me' during real interactions, as demonstrated in a full dialog example.
• Human participants on MTurk reported no significant difference in perceived task ease or understanding, likely due to the simplicity of the domain and lower user investment in MTurk tasks.
• The dual-track POMDP controller enabled the agent to balance dialog progression and KB augmentation, reducing redundant clarification questions and improving overall efficiency.
• In simulation and real-world evaluations, the agent outperformed static KB baselines in both accuracy and dialog efficiency, with measurable improvements in user satisfaction metrics.