[Paper Review] Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control

Mehrshad Zandigohar, Mo Han+9 more|arXiv (Cornell University)|Apr 8, 2021

EEG and Brain-Computer Interfaces4 citations

TL;DR

This paper proposes a Bayesian evidence fusion framework that combines electromyography (EMG) and vision data—using eye-gaze and RGB video—to improve grasp intent inference in prosthetic hand control. By integrating neural network-based EMG and visual classifiers with temporal phase segmentation, the method achieves 95.3% average accuracy in grasp classification, outperforming individual modalities by 13.66% (EMG) and 14.8% (vision) during the critical reaching phase.

ABSTRACT

Objective: For transradial amputees, robotic prosthetic hands promise to regain the capability to perform daily living activities. Current control methods based on physiological signals such as electromyography (EMG) are prone to yielding poor inference outcomes due to motion artifacts, muscle fatigue, and many more. Vision sensors are a major source of information about the environment state and can play a vital role in inferring feasible and intended gestures. However, visual evidence is also susceptible to its own artifacts, most often due to object occlusion, lighting changes, etc. Multimodal evidence fusion using physiological and vision sensor measurements is a natural approach due to the complementary strengths of these modalities. Methods: In this paper, we present a Bayesian evidence fusion framework for grasp intent inference using eye-view video, eye-gaze, and EMG from the forearm processed by neural network models. We analyze individual and fused performance as a function of time as the hand approaches the object to grasp it. For this purpose, we have also developed novel data processing and augmentation techniques to train neural network components. Results: Our results indicate that, on average, fusion improves the instantaneous upcoming grasp type classification accuracy while in the reaching phase by 13.66% and 14.8%, relative to EMG (81.64% non-fused) and visual evidence (80.5% non-fused) individually, resulting in an overall fusion accuracy of 95.3%. Conclusion: Our experimental data analyses demonstrate that EMG and visual evidence show complementary strengths, and as a consequence, fusion of multimodal evidence can outperform each individual evidence modality at any given time.

Motivation & Objective

• Address the limitations of single-modality control in prosthetic hands, such as motion artifacts and signal drift in EMG and occlusion/lighting issues in vision.
• Improve robustness and accuracy of grasp intent inference during the reaching phase, where decision timing is critical for robotic actuation.
• Develop a fusion framework that leverages complementary strengths of EMG and visual data to enhance classification performance.
• Introduce novel data processing and augmentation techniques, including copy-paste augmentation and temporal phase segmentation, to improve model generalization.
• Demonstrate that multimodal fusion consistently outperforms individual modalities across all phases, especially during the reaching phase.

Proposed method

• Employ a Bayesian evidence fusion framework to combine EMG and visual evidence using maximum likelihood estimation.
• Use a convolutional neural network (CNN) to classify grasp types from eye-view RGB video and eye-gaze data, with background generalization via copy-paste augmentation.
• Implement a separate neural network for EMG signal classification, including phase detection (rest, reach, grasp, return) to preserve temporal dynamics.
• Segment each grasp sequence into distinct temporal phases to avoid confusion between muscle activity patterns during different stages.
• Apply temporal smoothing to fused decisions, increasing accuracy to 96.8% by leveraging past decisions and system constraints.
• Train and evaluate models on a synchronized EMG and vision dataset collected from lower-limb amputees performing daily object interactions.

Experimental results

Research questions

• RQ1 How does multimodal fusion of EMG and vision improve grasp intent classification accuracy compared to individual modalities?
• RQ2 What is the impact of temporal phase segmentation on EMG-based grasp classification performance?
• RQ3 In which phase of the reaching-grasping cycle does fusion provide the greatest performance gain?
• RQ4 Can visual and EMG modalities compensate for each other’s limitations, such as occlusion or low EMG signal during rest?
• RQ5 To what extent does decision smoothing improve the robustness and accuracy of the fused grasp prediction?

Key findings

• Fusion of EMG and visual evidence improved grasp classification accuracy by 13.66% during the reaching phase compared to EMG alone.
• Fusion improved accuracy by 14.8% compared to vision alone during the reaching phase, demonstrating significant performance gains.
• The overall fusion accuracy reached 95.3%, with the highest accuracy (95.3%) recorded in the reaching phase, the most critical for control decisions.
• The visual classifier outperformed EMG during rest phases (90.69% vs. 16.86%), highlighting complementary strengths across phases.
• Even when both modalities provided strong evidence, fusion further improved accuracy, indicating added robustness beyond individual modality performance.
• Applying decision smoothing to fused outputs increased average accuracy to 96.8%, suggesting temporal consistency enhances real-world usability.