[Paper Review] Autonomy Infused Teleoperation with Application to BCI Manipulation

Katharina Muelling, Arun Venkatraman+8 more|arXiv (Cornell University)|Mar 18, 2015

EEG and Brain-Computer InterfacesNeuroscience41 references18 citations

TL;DR

This paper proposes a shared-control teleoperation framework that integrates computer vision, user intent inference, and adaptive arbitration between human input and autonomous control to enhance BCI-driven robotic manipulation. By using capture envelopes for grasp inference and adjustable assistance levels, the system significantly improves task success rates and reduces user effort in high-precision, cluttered, and novel-object scenarios with intracortical BCI users.

ABSTRACT

Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain-Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operator's capabilities and feelings of comfort and control while compensating for a task's difficulty. We present experimental results demonstrating significant performance improvement using the shared-control assistance framework on adapted rehabilitation benchmarks with two subjects implanted with intracortical brain-computer interfaces controlling a seven degree-of-freedom robotic manipulator as a prosthetic. Our results further indicate that shared assistance mitigates perceived user difficulty and even enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with novel objects in densely cluttered environments.

Motivation & Objective

• To address the challenges of low-bandwidth, noisy, and erratic neural commands in BCI-controlled robotic manipulation.
• To reduce user workload and perceived task difficulty in high-precision teleoperation tasks.
• To enable successful performance on previously infeasible tasks through intelligent autonomy infusion.
• To extend the system's applicability to real-world quality-of-life tasks like pouring liquids and opening doors.
• To develop a flexible, extensible architecture that supports novel object manipulation without prior model knowledge.

Proposed method

• The system uses depth-image template matching with RANSAC and ICP to recognize and localize objects in real-time using a 3D model library.
• It introduces 'capture envelopes'—dynamic, object- and grasp-specific regions of interest—for continuous and smooth grasp intention inference from low-dimensional user motion commands.
• User intent is inferred using a maximum entropy formulation that estimates the most likely goal based on motion trajectories and environmental context.
• A human-robot arbitration module blends user commands with autonomous motion plans using adjustable assistance levels to balance control and comfort.
• Compliant control is achieved via joint-stall detection and Jacobian transpose control using a wrist-mounted force-torque sensor, enabling safe environmental interaction.
• The framework is extended to novel objects using supervoxel segmentation and forward simulation of simplified hand models to detect feasible grasp points.

Experimental results

Research questions

• RQ1 Can a shared-control framework improve task performance and reduce user effort in BCI-controlled robotic manipulation?
• RQ2 How does adjustable assistance level affect user perception of control and task difficulty?
• RQ3 Can the system successfully handle complex, real-world tasks such as pouring liquids or opening doors with only low-dimensional BCI inputs?
• RQ4 To what extent can the system generalize to novel, unmodeled objects without prior 3D models?
• RQ5 How does integrating computer vision and intent inference enhance the robustness of BCI teleoperation in cluttered, dynamic environments?

Key findings

• The shared-control framework enabled two intracortical BCI users to successfully complete tasks that were previously impossible with direct teleoperation.
• In a pouring task, the system achieved 100% success rate across 10 trials, with only one trial failing to return the object to the table.
• The system significantly reduced perceived user difficulty, allowing subjects to perform high-precision tasks with less cognitive load.
• The use of capture envelopes improved grasp inference accuracy and smoothness compared to fixed-threshold methods.
• The architecture successfully extended to novel objects in dense clutter using model-free perception and grasp detection, validated as a proof-of-concept with a motion controller.
• The integration of autonomous assistance with BCI input enabled reliable manipulation in complex, real-world scenarios such as door opening and liquid pouring.