[Paper Review] Where to look first? Behaviour control for fetch-and-carry missions of service robots

Markus Bajones, Daniel Wolf+2 more|arXiv (Cornell University)|Oct 6, 2015

Robotic Path Planning Algorithms18 references3 citations

TL;DR

This paper proposes a behavior controller for service robots that uses semantic segmentation to autonomously identify high-probability search locations for fetch-and-carry tasks, eliminating the need for manually annotated positions. By analyzing scene context (e.g., tables, shelves), the system generates intelligent search poses, achieving comparable efficiency to pre-labeled locations while improving flexibility and human-oriented perception in domestic environments.

ABSTRACT

This paper presents the behaviour control of a service robot for intelligent object search in a domestic environment. A major challenge in service robotics is to enable fetch-and-carry missions that are satisfying for the user in terms of efficiency and human-oriented perception. The proposed behaviour controller provides an informed intelligent search based on a semantic segmentation framework for indoor scenes and integrates it with object recognition and grasping. Instead of manually annotating search positions in the environment, the framework automatically suggests likely locations to search for an object based on contextual information, e.g. next to tables and shelves. In a preliminary set of experiments we demonstrate that this behaviour control is as efficient as using manually annotated locations. Moreover, we argue that our approach will reduce the intensity of labour associated with programming fetch-and-carry tasks for service robots and that it will be perceived as more human-oriented.

Motivation & Objective

• To develop a behavior control system that enables service robots to perform fetch-and-carry tasks more flexibly and human-intuitively in domestic environments.
• To eliminate the need for manually annotating object search locations, reducing programming overhead and increasing adaptability to environmental changes.
• To improve robot behavior legibility and user satisfaction by prioritizing contextually likely object locations through semantic scene understanding.
• To evaluate the feasibility and performance of the proposed approach in real-world indoor settings using a robotic platform.

Proposed method

• The system uses SMACH, a Python-based hierarchical state machine framework, to orchestrate robot behaviors including perception, navigation, and grasping.
• Semantic segmentation of RGB-D data identifies potential object locations such as tables, shelves, and window sills based on scene context.
• The robot performs a 30° rotation and point cloud acquisition upon room entry to enable 3D scene understanding and search pose generation.
• Search positions are computed automatically from segmented surfaces, avoiding manual labeling while maintaining coverage of likely object locations.
• Object detection and recognition are performed using the upper camera, with grasping executed via a robotic arm equipped with a 2-finger gripper.
• The system integrates navigation, perception, and manipulation into a single behavior control pipeline using ROS and SMACH.

Experimental results

Research questions

• RQ1 Can a robot autonomously generate high-probability search locations for object retrieval without manual annotation?
• RQ2 How does the performance of semantic segmentation-based search compare to pre-labeled search positions in terms of time and success rate?
• RQ3 To what extent does the proposed method improve user perception of robot behavior in terms of intelligence and legibility?
• RQ4 What are the main performance bottlenecks in the current implementation, and how can they be mitigated?

Key findings

• The semantic segmentation-based approach achieved comparable search efficiency to manually annotated locations, with a success rate of 80% across test runs.
• The system required up to 2.5 times longer than pre-labeled searches due to broader search coverage, but this was expected due to increased flexibility.
• The highest search duration was 23:07 minutes for a wallet search, with 36 search positions, indicating a trade-off between coverage and speed.
• False positives occurred in 1 out of 4 runs (handbag on nightstand), suggesting room for improvement in recognition reliability.
• The system's main time costs stemmed from sequential semantic segmentation and 30° rotation, which could be reduced by parallelizing these steps, potentially saving up to 4 minutes in multi-room environments.
• Clustering nearby search poses could reduce the number of positions by up to 50%, improving efficiency without sacrificing coverage.