[Paper Review] NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities

Ruohan Zhang, Sharon Lee+12 more|arXiv (Cornell University)|Nov 2, 2023

EEG and Brain-Computer Interfaces8 citations

TL;DR

NOIR demonstrates a general-purpose brain-robot interface that uses non-invasive EEG to command robots for 20 everyday tasks, with a modular intention decoding pipeline and few-shot learning for efficiency.

ABSTRACT

We present Neural Signal Operated Intelligent Robots (NOIR), a general-purpose, intelligent brain-robot interface system that enables humans to command robots to perform everyday activities through brain signals. Through this interface, humans communicate their intended objects of interest and actions to the robots using electroencephalography (EEG). Our novel system demonstrates success in an expansive array of 20 challenging, everyday household activities, including cooking, cleaning, personal care, and entertainment. The effectiveness of the system is improved by its synergistic integration of robot learning algorithms, allowing for NOIR to adapt to individual users and predict their intentions. Our work enhances the way humans interact with robots, replacing traditional channels of interaction with direct, neural communication. Project website: https://noir-corl.github.io/.

Motivation & Objective

• Motivate the need for general-purpose brain-robot interfaces that go beyond task-specific systems.
• Develop a modular decoding pipeline that breaks down intent into what, how, and where signals from EEG.
• Equip robots with a library of parameterized primitive skills to execute high-level goals.
• Incorporate few-shot imitation learning to adapt to individual users and improve efficiency.

Proposed method

• Decode object selection from SSVEP using CCA with object segmentation via OWL-ViT.
• Decode interaction method and parameters from motor imagery using CSP+QDA with limited calibration data.
• Integrate a safety mechanism using EMG to confirm or interrupt decodings.
• Represent robot actions as parameterized primitive skills and implement them on a Panda arm and Tiago robot.
• Use retrieval-based few-shot learning with R3M features to predict object-skill choices from memories.
• Apply one-shot parameter learning with DINOv2-based semantic matching to predict cursor target points.

Figure 1: NOIR is a general-purpose brain-robot interface that allows humans to use their brain signals (1) to control robots to perform daily activities, such as making Sukiyaki (2), ironing clothes (7), playing Tic-Tac-Toe with friends (17), and petting a robot dog (21).

Experimental results

Research questions

• RQ1 Can NOIR achieve broad generalization to 20 diverse daily activities using EEG decoding?
• RQ2 How accurate are the decoding stages (What, How, Where) and the safety confirmation in practice?
• RQ3 Do few-shot learning components improve task efficiency and reduce human effort?
• RQ4 Can robots adapt to individual users with limited calibration data and still complete complex tasks?

Key findings

• NOIR achieves high object-selection decoding accuracy (SSVEP with 4-way classification at 81.2%).
• MI-based 2-way parameter selection (How) reaches 73.9% task-time accuracy; 4-way skill selection (What) ~42.2%.
• Overall decoding accuracy supports safe operation with a 100% safety-interruption mechanism in tested tasks.
• Object-skill retrieval learning reduces selection time by about 60% (from 45.7s to 18.1s).
• One-shot parameter learning using DINOv2 enables accurate cursor target prediction, reducing required cursor movement by ~41%.
• System demonstrates successful completion of 20 tasks across tabletop and mobile manipulation with few human interventions.

Figure 2: NOIR has two components, a modular pipeline for decoding goals from human brain signals, and a robotic system with a library of primitive skills. The robots possess the ability to learn to predict human intended goals hence reducing the human effort required for decoding.