[Paper Review] Kinematically-Informed Interactive Perception: Robot-Generated 3D Models for Classification

Abhishek Venkataraman, Brent Griffin+1 more|arXiv (Cornell University)|Jan 17, 2019

Robotics and Sensor-Based Localization32 references4 citations

TL;DR

This paper proposes Kinematically-Informed Interactive Perception (KIIP), a framework that uses a robot's kinematic knowledge during active manipulation to generate accurate 3D voxel models from RGBD data, enabling real-time classification of unknown objects in cluttered environments. By combining robot pose data with multi-view observations, KIIP achieves 80% classification accuracy on household objects and 60% on novel LEGO structures, demonstrating effective zero-shot and few-shot learning for real-world robotics applications.

ABSTRACT

To be useful in everyday environments, robots must be able to observe and learn about objects. Recent datasets enable progress for classifying data into known object categories; however, it is unclear how to collect reliable object data when operating in cluttered, partially-observable environments. In this paper, we address the problem of building complete 3D models for real-world objects using a robot platform, which can remove objects from clutter for better classification. Furthermore, we are able to learn entirely new object categories as they are encountered, enabling the robot to classify previously unidentifiable objects during future interactions. We build models of grasped objects using simultaneous manipulation and observation, and we guide the processing of visual data using a kinematic description of the robot to combine observations from different view-points and remove background noise. To test our framework, we use a mobile manipulation robot equipped with an RGBD camera to build voxelized representations of unknown objects and then classify them into new categories. We then have the robot remove objects from clutter to manipulate, observe, and classify them in real-time.

Motivation & Objective

• Address the challenge of reliable 3D object modeling and classification in cluttered, partially-observable real-world environments where traditional passive perception fails.
• Enable robots to learn and classify entirely new object categories not present in existing training datasets, such as those in ModelNet40.
• Leverage robot kinematics and active manipulation to improve 3D reconstruction and classification by combining multi-view observations with pose information.
• Develop a practical, real-time framework for mobile manipulation robots to isolate, observe, and classify unknown objects from dense clutter using RGBD sensing.
• Demonstrate that kinematically-informed fusion of visual data improves classification performance over passive or non-kinematically guided methods.

Proposed method

• Use a mobile manipulation robot (HSR) equipped with an RGBD camera to perform simultaneous manipulation and observation of grasped objects.
• Apply kinematic information from joint encoders to align and fuse multiple 2.5D point cloud observations into a single 3D voxel grid representation.
• Construct a KIIP-based 3D model by summing occupancy grids from multiple viewpoints during object rotation and repositioning.
• Train a shallow 3D convolutional neural network (CNN) on KIIP-generated 3D models to classify objects into known or newly learned categories.
• Use a two-stage training approach: pre-train on ModelNet40, then fine-tune on robot-collected KIIP data for novel object categories.
• Implement real-time object removal from clutter, followed by KIIP-based observation and classification to simulate interactive learning in unstructured environments.

Experimental results

Research questions

• RQ1 Can a robot generate reliable 3D object models in cluttered, partially-observable environments using active manipulation and visual sensing?
• RQ2 How effective is kinematically-informed fusion of multi-view observations in improving 3D reconstruction and object classification accuracy?
• RQ3 Can robots learn to classify entirely new object categories not seen during pre-training using self-generated KIIP data?
• RQ4 What is the performance of KIIP-based classification compared to passive perception or non-kinematically guided active perception?
• RQ5 How does the number of KIIP grasps influence classification accuracy for novel, unconventional objects like LEGO structures?

Key findings

• The KIIP framework achieved 80% classification accuracy on household objects using a network trained solely on robot-collected KIIP data, demonstrating the feasibility of self-supervised 3D object learning.
• For novel LEGO structures, the OL-E2E network achieved 60% overall classification accuracy across all training configurations, outperforming other networks in robustness and speed.
• The OL-E2E network required only 14 seconds of training time on a single GPU, making it the fastest and most efficient among the tested models.
• The FG-OL and OL-E2E networks, both trained on KIIP data, significantly outperformed the FG network (trained only on ModelNet40) on novel LEGO objects, indicating the value of robot-generated data for few-shot learning.
• The method successfully classified unknown objects in dense clutter by removing them one-by-one, observing via KIIP, and assigning them to newly learned categories.
• The use of kinematic information enabled accurate alignment and fusion of 2.5D point clouds without relying on feature matching or complex 3D registration, simplifying the perception pipeline.