[Paper Review] Spiking Neural Networks for Early Prediction in Human Robot Collaboration

Tian Zhou, Juan Wachs|arXiv (Cornell University)|Jul 29, 2018

Action Observation and Synchronization55 references5 citations

TL;DR

This paper proposes the Turn-Taking Spiking Neural Network (TTSNet), a spiking neural network model that predicts human turn-taking intentions in human-robot collaboration using multimodal cues (physical, neurological, and physiological). It achieves an F1 score of 0.894 at 100% action completion and surpasses human performance when given less than 40% of the action, enabling proactive robotic assistance in surgical settings.

ABSTRACT

This paper introduces the Turn-Taking Spiking Neural Network (TTSNet), which is a cognitive model to perform early turn-taking prediction about human or agent's intentions. The TTSNet framework relies on implicit and explicit multimodal communication cues (physical, neurological and physiological) to be able to predict when the turn-taking event will occur in a robust and unambiguous fashion. To test the theories proposed, the TTSNet framework was implemented on an assistant robotic nurse, which predicts surgeon's turn-taking intentions and delivers surgical instruments accordingly. Experiments were conducted to evaluate TTSNet's performance in early turn-taking prediction. It was found to reach a F1 score of 0.683 given 10% of completed action, and a F1 score of 0.852 at 50% and 0.894 at 100% of the completed action. This performance outperformed multiple state-of-the-art algorithms, and surpassed human performance when limited partial observation is given (< 40%). Such early turn-taking prediction capability would allow robots to perform collaborative actions proactively, in order to facilitate collaboration and increase team efficiency.

Motivation & Objective

• To develop a cognitive model that enables robots to anticipate human turn-taking in collaborative tasks.
• To integrate multimodal cues—physical, neurological, and physiological—for robust and unambiguous prediction of human intentions.
• To evaluate the performance of the proposed model in real-time human-robot collaboration, particularly in surgical assistance scenarios.
• To demonstrate that the model outperforms both existing algorithms and human experts under partial observation conditions.

Proposed method

• The TTSNet framework employs spiking neural networks to model temporal dynamics in human-robot interaction.
• It integrates implicit and explicit multimodal signals, including motion, gaze, and physiological signals such as EMG or EEG.
• The network is trained to predict the timing of turn-taking events based on partial action sequences.
• The model uses a spiking neuron architecture to process temporal information efficiently and mimic biological neural processing.
• The system is evaluated in a robotic nurse scenario where it predicts surgeon instrument requests during surgical tasks.
• Performance is measured using F1 score at different stages of action completion (10%, 50%, 100%).

Experimental results

Research questions

• RQ1 Can a spiking neural network effectively predict human turn-taking intentions in human-robot collaboration using multimodal cues?
• RQ2 How early can the TTSNet model predict turn-taking events with high accuracy under partial observation?
• RQ3 Does the TTSNet model outperform state-of-the-art algorithms and human experts in early prediction tasks?
• RQ4 What is the impact of integrating neurological and physiological signals on prediction robustness and accuracy?

Key findings

• TTSNet achieved an F1 score of 0.683 when predicting turn-taking with only 10% of the action completed.
• The F1 score improved to 0.852 at 50% of action completion and reached 0.894 at 100% completion.
• The model surpassed human performance in early prediction when given less than 40% of the action sequence.
• The integration of multimodal cues (physical, neurological, and physiological) enhanced prediction robustness and accuracy.