[Paper Review] Programming by Demonstration with User-Specified Perceptual Landmarks

Justin Huang, Maya Çakmak|arXiv (Cornell University)|Dec 2, 2016

Robot Manipulation and Learning22 references3 citations

TL;DR

This paper introduces a flexible, user-driven perception system for Programming by Demonstration (PbD) that allows non-expert users to define custom 3D perceptual landmarks using a point cloud interface. By capturing user-specified point cloud patches—including parts of objects, scene features, or empty spaces—the system enables robust localization of these landmarks in new environments using a modified ICP algorithm with empty space constraints, significantly improving generalization of manipulation tasks across diverse, non-tabletop, and cluttered scenes without requiring pre-existing object models or fiducials.

ABSTRACT

Programming by demonstration (PbD) is an effective technique for developing complex robot manipulation tasks, such as opening bottles or using human tools. In order for such tasks to generalize to new scenes, the robot needs to be able to perceive objects, object parts, or other task-relevant parts of the scene. Previous work has relied on rigid, task-specific perception systems for this purpose. This paper presents a flexible and open-ended perception system that lets users specify perceptual "landmarks" during the demonstration, by capturing parts of the point cloud from the demonstration scene. We present a method for localizing landmarks in new scenes and experimentally evaluate this method in a variety of settings. Then, we provide examples where user-specified landmarks are used together with PbD on a PR2 robot to perform several complex manipulation tasks. Finally, we present findings from a user evaluation of our landmark specification interface demonstrating its feasibility as an end-user tool.

Motivation & Objective

• To overcome the limitations of rigid, task-specific perception systems in existing PbD frameworks that restrict landmark types and generalization to tabletop or clutter-free scenes.
• To enable end-users to define task-relevant perceptual landmarks—such as object parts, scene features, or empty spaces—without requiring CAD models or fiducials.
• To develop a perception system that can robustly localize user-defined landmarks in new, unseen scenes, even in cluttered or unconventional environments.
• To evaluate the feasibility of the landmark specification interface for novice users in real-world manipulation tasks.
• To demonstrate that custom landmarks enable complex manipulation tasks (e.g., picking from shelves, using tools) to generalize across diverse environments beyond standard tabletop setups.

Proposed method

• Users define custom landmarks by capturing 3D point cloud patches using a 3D bounding box interface that allows adjustment of all six sides to select a region of interest from the robot’s sensor data.
• The system represents landmarks as 3D point clouds with explicit modeling of surrounding empty space (negative space), enhancing uniqueness and localization robustness.
• A modified Iterative Closest Point (ICP) algorithm is used to register the user-defined landmark point cloud to candidate regions in a new scene, incorporating geometric constraints and empty space to improve accuracy.
• The algorithm performs iterative closest point matching while penalizing misalignments in both surface geometry and expected empty space, increasing discriminability in ambiguous or cluttered scenes.
• The system integrates with a PbD pipeline where end-effector poses are recorded relative to the localized landmarks, enabling task execution in new environments.
• A usability study evaluates the interface with non-expert users, measuring time to define landmarks and identifying interface limitations through qualitative feedback.

Experimental results

Research questions

• RQ1 Can user-specified perceptual landmarks, defined via point cloud patches, be reliably localized in new, unseen scenes using a perception algorithm that accounts for both surface geometry and empty space?
• RQ2 To what extent do custom landmarks improve the generalization of PbD-based manipulation tasks across non-tabletop, cluttered, or unconventional environments compared to rigid, pre-defined perception systems?
• RQ3 How feasible is the landmark specification interface for non-expert users in real-world PbD scenarios, and what usability challenges emerge during the creation of task-relevant landmarks?
• RQ4 In what ways do custom landmarks enable manipulation tasks that would otherwise require task-specific perception code or fiducial markers?
• RQ5 How do limitations in sensor view, occlusion, or landmark uniqueness affect the robustness of landmark localization and task execution?

Key findings

• The system successfully localized user-specified landmarks in 14 out of 16 test scenarios across diverse environments, including non-tabletop settings like shelves, doors, and cluttered scenes.
• The addition of empty space modeling in the landmark representation significantly improved localization accuracy in ambiguous or cluttered scenes, reducing false positives.
• The usability study showed that novice users could define viable landmarks in an average of 1 minute 11 seconds (SD = 42 seconds), with all participants successfully completing at least one task.
• Failure cases were primarily due to sensor limitations (e.g., partial depth data from angled surfaces) or landmark ambiguity (e.g., flat surfaces mistaken for empty spots), indicating the system’s dependence on landmark distinctiveness.
• Tasks such as picking bowls from a shelf, using a hanging tool, or distinguishing trash bins from similar containers were successfully executed using custom landmarks, demonstrating generalization beyond tabletops.
• The bounding box interface was functional but identified as improvable, with users requesting separate translation controls or an automatic initial guess feature to reduce cognitive load.