[Paper Review] Incremental Learning for Robot Perception through HRI

Sepehr Valipour, Camilo Perez+1 more|arXiv (Cornell University)|Jan 17, 2017

Multimodal Machine Learning Applications14 references4 citations

TL;DR

This paper proposes a human-robot interaction (HRI)-driven incremental learning framework that enables robots to continuously improve visual perception through active human guidance. By combining deep learning-based object detection and recognition with real-time human feedback via pointing and voice commands, the robot incrementally expands its knowledge base, achieving stable performance (top-1 accuracy >0.45) even after adding numerous new object classes without catastrophic forgetting.

ABSTRACT

Scene understanding and object recognition is a difficult to achieve yet crucial skill for robots. Recently, Convolutional Neural Networks (CNN), have shown success in this task. However, there is still a gap between their performance on image datasets and real-world robotics scenarios. We present a novel paradigm for incrementally improving a robot's visual perception through active human interaction. In this paradigm, the user introduces novel objects to the robot by means of pointing and voice commands. Given this information, the robot visually explores the object and adds images from it to re-train the perception module. Our base perception module is based on recent development in object detection and recognition using deep learning. Our method leverages state of the art CNNs from off-line batch learning, human guidance, robot exploration and incremental on-line learning.

Motivation & Objective

• To address the challenge of robot perception in real-world environments where object categories are dynamic and not fully known in advance.
• To enable robots to incrementally learn new object categories through natural human-robot interaction, such as pointing and verbal labeling.
• To develop a system that maintains high recognition accuracy while continuously adding new classes, avoiding catastrophic forgetting.
• To establish a common ground of knowledge between humans and robots through interactive, discourse-based learning.
• To demonstrate the feasibility of incremental visual perception in real-world robotic applications, such as assisting in electronics workshops.

Proposed method

• The system uses a pre-trained deep learning model (based on RPN and CNN) for real-time object detection and recognition, with inference time of 150ms on a GeForce 960 GPU.
• New object classes are introduced via human interaction: users point to an object and label it verbally, establishing a common ground.
• The robot autonomously explores the object using its onboard camera, collecting multiple images for retraining the perception module.
• The model is incrementally fine-tuned using the newly collected images, updating the classifier head to include new categories.
• The approach leverages off-the-shelf state-of-the-art CNNs for initial perception, then applies online incremental learning to adapt to new data.
• Performance is evaluated using top-1 accuracy and Average Precision (AP), with metrics computed over test sets containing both old and new classes.

Experimental results

Research questions

• RQ1 Can a robot incrementally expand its visual perception capability through continuous human interaction without forgetting previously learned classes?
• RQ2 How does the recognition accuracy of a deep learning model degrade when new object classes are added incrementally, and can this degradation be minimized?
• RQ3 To what extent do images collected by the robot through human-guided exploration perform compared to generic datasets like ImageNet in incremental learning scenarios?
• RQ4 Can natural human-robot interaction (pointing and voice commands) effectively serve as a scalable and reliable method for teaching new object categories to a robot?
• RQ5 How does the integration of temporal consistency in object detection and recognition improve long-term perception robustness?

Key findings

• The system maintains a high level of recognition performance, achieving a top-1 accuracy of 0.45 on the baseline model, which is expected given the difficulty of the MS-COCO dataset.
• After incrementally adding new object classes, the model shows only a low-slope decline in accuracy, indicating minimal catastrophic forgetting even after multiple additions.
• The performance of images collected by the robot through HRI is nearly equivalent to that of images from ImageNet, with only a slight decline in accuracy, suggesting the robot-collected data is highly effective for incremental learning.
• The system achieves real-time inference at 150ms per frame on a consumer-grade GPU, outperforming R-CNN in speed while maintaining competitive Average Precision (0.2026) compared to the state of the art (0.224).
• The combination of human guidance and robot exploration enables reliable and scalable learning of new object categories, as demonstrated in a real-world electronics workshop scenario.
• The total system accuracy—requiring both correct localization (IoU > 0.5) and correct classification—reaches 0.1704, indicating that localization and recognition are both challenging but manageable in the incremental setting.