[Paper Review] Bootstrapping Robotic Ecological Perception from a Limited Set of Hypotheses Through Interactive Perception

Léni K. Le Goff, Ghanim Mukhtar+2 more|arXiv (Cornell University)|Jan 30, 2019

Visual Attention and Saliency DetectionComputer Science48 references3 citations

TL;DR

This paper proposes an interactive perception framework that enables a robot to bootstrap ecological perception by learning to identify moveable parts in unstructured environments without prior assumptions about scene structure. Using online training of a collaborative mixture models classifier and uncertainty-based exploration, the robot builds a relevance map indicating potentially interactive regions, validated on simulations and a PR2 robot with robust performance across complex scenarios.

ABSTRACT

To solve its task, a robot needs to have the ability to interpret its perceptions. In vision, this interpretation is particularly difficult and relies on the understanding of the structure of the scene, at least to the extent of its task and sensorimotor abilities. A robot with the ability to build and adapt this interpretation process according to its own tasks and capabilities would push away the limits of what robots can achieve in a non controlled environment. A solution is to provide the robot with processes to build such representations that are not specific to an environment or a situation. A lot of works focus on objects segmentation, recognition and manipulation. Defining an object solely on the basis of its visual appearance is challenging given the wide range of possible objects and environments. Therefore, current works make simplifying assumptions about the structure of a scene. Such assumptions reduce the adaptivity of the object extraction process to the environments in which the assumption holds. To limit such assumptions, we introduce an exploration method aimed at identifying moveable elements in a scene without considering the concept of object. By using the interactive perception framework, we aim at bootstrapping the acquisition process of a representation of the environment with a minimum of context specific assumptions. The robotic system builds a perceptual map called relevance map which indicates the moveable parts of the current scene. A classifier is trained online to predict the category of each region (moveable or non-moveable). It is also used to select a region with which to interact, with the goal of minimizing the uncertainty of the classification. A specific classifier is introduced to fit these needs: the collaborative mixture models classifier. The method is tested on a set of scenarios of increasing complexity, using both simulations and a PR2 robot.

Motivation & Objective

• Address the challenge of robotic perception in unstructured environments by minimizing reliance on pre-defined scene hypotheses such as tabletop assumptions.
• Enable a robot to autonomously learn which parts of a scene are interactive (e.g., moveable) through physical interaction, rather than relying on pre-segmented object hypotheses.
• Develop a perception system that adapts to the robot’s sensorimotor capabilities and task context, avoiding human-crafted assumptions about object shapes or layouts.
• Bootstrap a general-purpose perceptual map—called a relevance map—that identifies regions likely to afford manipulation actions like pushing.
• Enable online, incremental learning of a classifier that improves over time through interaction, reducing uncertainty and balancing class representation.

Proposed method

• The robot uses a push primitive to interact with selected regions of the environment, observing whether the scene changes after the action.
• A change detector algorithm determines whether a region is moveable by comparing pre- and post-action visual observations.
• A collaborative mixture models (CMM) classifier is trained online using interaction data, with a focus on handling non-linearly separable data and estimating classification uncertainty.
• An uncertainty-based sampling strategy selects the next region to interact with, prioritizing areas where the classifier is most uncertain to improve learning efficiency.
• The classifier is trained incrementally, allowing the system to adapt to new environments without retraining from scratch.
• A relevance map is computed from the classifier’s output, indicating the probability that each image region is moveable, guiding future exploration.

Experimental results

Research questions

• RQ1 Can a robot learn to identify moveable parts in a scene without relying on scene-specific assumptions such as tabletops or object primitives?
• RQ2 How can interactive perception be used to bootstrap a perceptual map that reflects the robot’s own sensorimotor capabilities and task goals?
• RQ3 What classifier architecture enables effective online learning with uncertainty estimation and minimal hyperparameter tuning in a real-world robotic setting?
• RQ4 To what extent can uncertainty-based exploration reduce the number of interactions needed to achieve high classification accuracy in complex visual scenes?
• RQ5 How does the relevance map generated by the system support downstream tasks like object discovery or manipulation planning?

Key findings

• The collaborative mixture models (CMM) classifier successfully handles non-linearly separable data and provides reliable uncertainty estimates, enabling effective active learning.
• The uncertainty-based sampling strategy significantly reduces the number of interactions required to achieve high classification accuracy by focusing on ambiguous regions.
• The method generalizes across diverse environments, including simulations and real-world scenarios with moveable balls, bricks, and complex cluttered scenes.
• The relevance map effectively identifies moveable regions, with performance robust to variations in feature space complexity and color descriptors.
• The approach reduces reliance on pre-defined scene hypotheses, enabling adaptation to new environments without retraining or manual reconfiguration.
• Experiments on a PR2 robot demonstrate the feasibility and robustness of the framework in real-world settings, with stable performance across multiple trials and increasing scenario complexity.