[Paper Review] Verbal Focus-of-Attention System for Learning-from-Observation

Naoki Wake, Iori Yanokura+2 more|arXiv (Cornell University)|Jul 17, 2020

Robot Manipulation and Learning59 references4 citations

TL;DR

This paper proposes a verbal focus-of-attention (FoA) system that enhances learning-from-observation (LfO) in robotics by using verbal instructions to guide spatio-temporal filtering of human demonstrations. By interpreting object names and attributes to define where-to-look filters and detecting grasp/release timings to create when-to-look filters, the system improves action localization robustness in cluttered, noisy environments compared to state-of-the-art vision-only methods.

ABSTRACT

The learning-from-observation (LfO) framework aims to map human demonstrations to a robot to reduce programming effort. To this end, an LfO system encodes a human demonstration into a series of execution units for a robot, which are referred to as task models. Although previous research has proposed successful task-model encoders, there has been little discussion on how to guide a task-model encoder in a scene with spatio-temporal noises, such as cluttered objects or unrelated human body movements. Inspired by the function of verbal instructions guiding an observer's visual attention, we propose a verbal focus-of-attention (FoA) system (i.e., spatio-temporal filters) to guide a task-model encoder. For object manipulation, the system first recognizes the name of a target object and its attributes from verbal instructions. The information serves as a where-to-look FoA filter to confine the areas in which the target object existed in the demonstration. The system then detects the timings of grasp and release that occurred in the filtered areas. The timings serve as a when-to-look FoA filter to confine the period of object manipulation. Finally, a task-model encoder recognizes the task models by employing FoA filters. We demonstrate the robustness of the verbal FoA in attenuating spatio-temporal noises by comparing it with an existing action localization network. The contributions of this study are as follows: (1) to propose a verbal FoA for LfO, (2) to design an algorithm to calculate FoA filters from verbal input, and (3) to demonstrate the effectiveness of a verbal FoA in localizing an action by comparing it with a state-of-the-art vision system.

Motivation & Objective

• Address the challenge of noisy human demonstrations in learning-from-observation (LfO), where cluttered objects and irrelevant body movements hinder accurate task modeling.
• Improve the robustness of task-model encoders in real-world scenarios with spatio-temporal noise by introducing a mechanism inspired by human attention guided by verbal instructions.
• Develop a system that translates natural language instructions into dynamic spatio-temporal filters to guide attention during demonstration encoding.
• Demonstrate that verbal FoA filters enhance localization accuracy of grasp and release actions in complex, noisy scenes compared to vision-only baselines.

Proposed method

• Parse verbal instructions to extract target object names and attributes, which are used to generate a 'where-to-look' spatial filter confining the region of interest in the demonstration.
• Detect grasp and release actions within the spatially filtered regions using temporal action localization, forming a 'when-to-look' temporal filter.
• Apply the combined spatio-temporal FoA filters to constrain the input to the task-model encoder, reducing sensitivity to irrelevant visual noise.
• Use a two-stage filtering process: first spatial filtering via object recognition and attribute parsing, then temporal filtering via action boundary detection.
• Integrate the FoA filters into a standard LfO pipeline, enabling the task-model encoder to focus only on relevant segments of the demonstration.
• Leverage existing action localization networks as baselines for comparison, while evaluating the impact of verbal FoA on localization accuracy and noise robustness.

Experimental results

Research questions

• RQ1 Can verbal instructions be effectively used to generate spatio-temporal filters that improve attention during learning-from-observation?
• RQ2 How does the verbal FoA system perform in localizing grasp and release actions under cluttered visual conditions compared to vision-only baselines?
• RQ3 To what extent does the FoA system reduce sensitivity to spatio-temporal noise such as irrelevant human movements and distractor objects?
• RQ4 What is the impact of combining object name and attribute recognition with temporal action detection in guiding attention for task modeling?
• RQ5 Can the proposed FoA mechanism be integrated into existing LfO frameworks without requiring major architectural changes?

Key findings

• The verbal FoA system significantly improves action localization accuracy in cluttered scenes by filtering out irrelevant visual noise through language-guided spatio-temporal constraints.
• The system achieves robust localization of grasp and release actions even when distractor objects or unrelated body movements are present in the demonstration.
• Compared to a state-of-the-art vision-only action localization network, the FoA-enhanced system demonstrates superior performance under high spatio-temporal noise conditions.
• The integration of verbal input into the attention mechanism enables more reliable and interpretable task-model extraction from human demonstrations.
• The proposed method effectively decouples the task of object localization from action boundary detection by using language to guide spatial and temporal filtering.
• The system's performance is validated through quantitative comparison, showing measurable gains in localization F1-score and robustness across multiple noisy demonstration scenarios.