[Paper Review] Solving Service Robot Tasks: UT Austin Villa@Home 2019 Team Report

Rishi Shah, Yuqian Jiang+9 more|arXiv (Cornell University)|Sep 14, 2019

Robotics and Automated Systems13 references4 citations

TL;DR

This paper presents the UT Austin Villa@Home 2019 team's approach to solving a broad range of autonomous service robot tasks using the Toyota HSR platform, integrating AI technologies like perception, navigation, manipulation, and human-robot interaction. Despite hardware issues preventing full competition demonstration, the team achieved significant progress in task execution, particularly in navigation, person following, and object placement, highlighting key challenges and future directions for robust, unified HRI systems.

ABSTRACT

RoboCup@Home is an international robotics competition based on domestic tasks requiring autonomous capabilities pertaining to a large variety of AI technologies. Research challenges are motivated by these tasks both at the level of individual technologies and the integration of subsystems into a fully functional, robustly autonomous system. We describe the progress made by the UT Austin Villa 2019 RoboCup@Home team which represents a significant step forward in AI-based HRI due to the breadth of tasks accomplished within a unified system. Presented are the competition tasks, component technologies they rely on, our initial approaches both to the components and their integration, and directions for future research.

Motivation & Objective

• To develop a unified, autonomous service robot system capable of executing diverse domestic tasks in the RoboCup@Home 2019 competition.
• To address the integration challenges of multiple AI subsystems—perception, navigation, manipulation, and HRI—into a robust, real-time system.
• To advance AI-based human-robot interaction by implementing responsive, context-aware behaviors such as person following and gesture recognition.
• To evaluate and document the performance of component technologies and their integration under real-world constraints, despite hardware limitations.
• To identify key research gaps and future directions for autonomous service robots in unstructured, human-centered environments.

Proposed method

• The team used the Toyota HSR robot with a holonomic base, 4-DoF arm, and RGBD head camera, augmented with an external GPU laptop for real-time deep learning inference.
• Perception tasks relied on YOLO for object detection, OpenPose for human keypoint estimation, and LIDAR/fused point cloud costmaps for obstacle and object localization.
• Navigation and path planning were based on local costmaps and A* planning, with motion to a target point calculated as the farthest non-colliding point along the shortest path to a person or object.
• Person following was implemented using a behavior tree framework that coordinated searching and tracking, with waving gestures used to identify targets.
• For manipulation tasks like storing groceries and cleaning tables, the robot used initial localization and then refined positioning by aligning with visible geometric features (e.g., dishwasher corners) using sensor fusion.
• The system used semantic reasoning to guide object placement (e.g., grouping similar items in the pantry) and employed gesture recognition to detect customer requests in the Restaurant task.

Experimental results

Research questions

• RQ1 How can a unified robotic system effectively integrate multiple AI components—perception, navigation, manipulation, and HRI—into a robust, autonomous service robot?
• RQ2 What are the key challenges in enabling real-time, responsive human-robot interaction in unstructured, dynamic environments such as a mock home?
• RQ3 How can robots reliably perform manipulation tasks like object placement and retrieval when precise localization is difficult and failure leads to disqualification?
• RQ4 To what extent can behavior trees and sensor fusion improve the reliability of navigation and target following in real-world settings?
• RQ5 What role does semantic understanding of human preferences (e.g., object grouping) play in improving task success and perceived naturalness in service robotics?

Key findings

• The team successfully demonstrated core capabilities in person following, gesture detection, and object placement, with the robot able to locate and align with the dishwasher using visual and LIDAR-based corner detection.
• The behavior tree-based system enabled responsive and reliable person following in uncrowded environments, though performance degraded in high-density or dynamic settings.
• The robot achieved accurate alignment with the dishwasher by fusing LIDAR and visual data, ensuring precise object placement and avoiding disqualification.
• The use of gesture-based detection (e.g., waving) allowed the robot to identify active customers in the Restaurant task, enabling mapless navigation to a safe proximity.
• Despite hardware failures preventing full competition performance, the team documented successful trials across all tasks, demonstrating the feasibility of their integrated approach.
• The study highlighted the need for improved person re-identification and sensor fusion techniques to enhance robustness in real-world HRI applications.