[Paper Review] Robot Language Learning, Generation, and Comprehension

Daniel Paul Barrett, Scott Alan Bronikowski+2 more|arXiv (Cornell University)|Aug 25, 2015

Topic Modeling23 references3 citations

TL;DR

This paper presents a unified framework for grounding natural language in robotic navigation, enabling robots to learn, generate, and comprehend language-based driving instructions through real-world teleoperation. Using human-annotated robot paths, the system acquires semantic meanings of nouns and prepositions, achieving 94.6% correctness and 85.6% completeness in human evaluations of generated descriptions and autonomous path execution.

ABSTRACT

We present a unified framework which supports grounding natural-language semantics in robotic driving. This framework supports acquisition (learning grounded meanings of nouns and prepositions from human annotation of robotic driving paths), generation (using such acquired meanings to generate sentential description of new robotic driving paths), and comprehension (using such acquired meanings to support automated driving to accomplish navigational goals specified in natural language). We evaluate the performance of these three tasks by having independent human judges rate the semantic fidelity of the sentences associated with paths, achieving overall average correctness of 94.6% and overall average completeness of 85.6%.

Motivation & Objective

• To enable robots to ground natural language semantics in physical robotic driving through real-world interaction.
• To learn the meanings of nouns and prepositions from human-annotated robot paths in continuous 2D space.
• To generate accurate natural-language descriptions of new manually driven robot paths.
• To autonomously drive a robot to fulfill navigational goals specified in natural language.
• To evaluate the semantic fidelity of generated descriptions and induced paths using human judges.

Proposed method

• A human teleoperator drives a mobile robot through various floorplans while providing sentential instructions, generating a corpus of path-sentence pairs.
• The system learns grounded meanings of nouns (e.g., bag, chair) and prepositions (e.g., left of, in front of) from this corpus using real-time odometry and inertial guidance to reconstruct paths.
• Learned semantic models are used to generate natural-language descriptions of new robot paths driven under teleoperation.
• The same models enable comprehension: interpreting natural-language commands to autonomously plan and execute corresponding robot paths.
• A human evaluation framework assesses correctness (truth of prepositional phrases) and completeness (coverage of path) using 3 independent judges per sample.
• The system is evaluated on 100 samples each for generation and comprehension, with additional comparisons using hand-constructed models.

Experimental results

Research questions

• RQ1 Can a robot learn the semantic meanings of nouns and prepositions from real-world robot paths paired with natural-language descriptions?
• RQ2 Can the learned semantics be used to generate accurate and complete natural-language descriptions of new robot driving paths?
• RQ3 Can the robot autonomously execute paths that satisfy natural-language navigational commands based on learned semantics?
• RQ4 How does the performance of the learned models compare to hand-designed models in terms of correctness and completeness?
• RQ5 What is the semantic fidelity of generated descriptions and induced paths in human evaluations?

Key findings

• The system achieved an average correctness rate of 94.6% in human evaluations of generated sentences relative to driven paths.
• The average completeness of generated descriptions was 85.6%, indicating strong coverage of the path.
• For comprehension, the system achieved 96.2% correctness on planned paths and 95.5% on measured paths, demonstrating high accuracy in path execution.
• Comprehension completeness averaged 88.5% on planned paths and 84.7% on measured paths, showing robust path coverage.
• The learned models performed nearly as well as hand-constructed models, with 92.0% correctness and 84.2% completeness in generation, confirming effective automatic learning.
• Judges rated sentence length as 'about right' in just over half of cases, with a tendency for generated sentences to be too long rather than too short.