[Paper Review] Find my mug: Efficient object search with a mobile robot using semantic segmentation

Daniel Wolf, Markus Bajones+2 more|arXiv (Cornell University)|Apr 23, 2014

Robotics and Sensor-Based Localization2 references3 citations

TL;DR

This paper presents an efficient 3D semantic segmentation framework using a Randomized Decision Forest (RDF) for mobile robots, enabling real-time indoor scene understanding. It achieves 71.7% class average accuracy and 77.2% global accuracy on the NYU Depth dataset, significantly accelerating object search by prioritizing likely locations such as tables.

ABSTRACT

In this paper, we propose an efficient semantic segmentation framework for indoor scenes, tailored to the application on a mobile robot. Semantic segmentation can help robots to gain a reasonable understanding of their environment, but to reach this goal, the algorithms not only need to be accurate, but also fast and robust. Therefore, we developed an optimized 3D point cloud processing framework based on a Randomized Decision Forest, achieving competitive results at sufficiently high frame rates. We evaluate the capabilities of our method on the popular NYU depth dataset and our own data and demonstrate its feasibility by deploying it on a mobile service robot, for which we could optimize an object search procedure using our results.

Motivation & Objective

• Address the challenge of real-time semantic segmentation for mobile service robots in indoor environments.
• Develop a fast and accurate segmentation framework suitable for deployment on resource-constrained robotic systems.
• Enable intelligent object search by leveraging semantic labels to prioritize likely object locations.
• Optimize the segmentation pipeline for high frame rates without sacrificing accuracy.
• Demonstrate the practical application of semantic segmentation in a real-world mobile robot scenario.

Proposed method

• Apply a Randomized Decision Forest (RDF) to classify 3D point cloud clusters derived from RGB-D data.
• Use oversegmentation via simple clustering to generate initial superpoints for feature extraction.
• Extract geometric and color features from each superpoint to train the RDF for semantic labeling.
• Apply a Conditional Random Field (CRF) to refine predictions using local spatial context.
• Optimize the RDF with hyperparameters such as number of trees and maximum tree depth for speed and accuracy.
• Deploy the framework on a mobile robot to guide object search by identifying likely object locations (e.g., tables).

Experimental results

Research questions

• RQ1 Can a fast and accurate semantic segmentation framework be developed for real-time deployment on mobile robots?
• RQ2 How does the RDF-based approach compare to existing methods in terms of accuracy and inference speed on indoor 3D scenes?
• RQ3 To what extent can semantic segmentation improve the efficiency of object search in service robotics?
• RQ4 What hyperparameter settings (e.g., number of trees, depth) yield optimal trade-offs between accuracy and speed?
• RQ5 Can semantic segmentation results be effectively used to guide dynamic, context-aware object search on a mobile robot?

Key findings

• The RDF-based framework achieves 71.7% class average accuracy and 77.2% global accuracy on the NYU Depth V2 dataset, outperforming prior work in terms of speed-accuracy trade-off.
• With 8 trees and a maximum depth of 8, the system processes 640×480 point clouds at approximately 1 frame per second on a standard laptop.
• The method significantly improves object search efficiency by identifying tables as primary search locations, reducing the search space.
• Increasing the number of trees improves accuracy up to 8 trees, after which performance saturates.
• Higher tree depth improves performance up to a point, but deeper trees do not improve results due to limited training data.
• The framework demonstrates feasibility in real-world deployment on a mobile service robot, enabling context-aware object search.