[Paper Review] Adaptive Grasp Control through Multi-Modal Interactions for Assistive Prosthetic Devices

Michelle Esponda, Thomas M. Howard|arXiv (Cornell University)|Oct 18, 2018

Muscle activation and electromyography studies28 references4 citations

TL;DR

This paper presents an adaptive prosthetic hand control system that learns user preferences through multi-modal inputs—myoelectric signals, vision, touch, and speech—enabling real-time grasp adaptation. By incorporating in-situ feedback via touchscreen and voice commands, the system improves grasp classification accuracy, achieving 78% probability for a novel banana object after 120 training examples, demonstrating personalized, interactive learning for prosthetic control.

ABSTRACT

The hand is one of the most complex and important parts of the human body. The dexterity provided by its multiple degrees of freedom enables us to perform many of the tasks of daily living which involve grasping and manipulating objects of interest. Contemporary prosthetic devices for people with transradial amputations or wrist disarticulation vary in complexity, from passive prosthetics to complex devices that are body or electrically driven. One of the important challenges in developing smart prosthetic hands is to create devices which are able to mimic all activities that a person might perform and address the needs of a wide variety of users. The approach explored here is to develop algorithms that permit a device to adapt its behavior to the preferences of the operator through interactions with the wearer. This device uses multiple sensing modalities including muscle activity from a myoelectric armband, visual information from an on-board camera, tactile input through a touchscreen interface, and speech input from an embedded microphone. Presented within this paper are the design, software and controls of a platform used to evaluate this architecture as well as results from experiments deigned to quantify the performance.

Motivation & Objective

• To address the challenge of personalized prosthetic control by enabling devices to adapt to individual user preferences.
• To reduce the cognitive load on users by minimizing the need for pre-mapped muscle signal patterns through interactive learning.
• To improve grasp classification accuracy for diverse and novel objects using multi-modal sensory feedback.
• To develop a real-time, on-device learning framework that integrates tactile and spoken feedback for continuous model refinement.
• To create a robust, user-adaptive prosthetic control system that supports long-term customization and usability.

Proposed method

• The system uses surface electromyography (sEMG) from a myoelectric armband to detect forearm muscle activity and infer intended grasp types.
• A convolutional neural network (CNN) processes visual input from an on-board camera to classify object shape and suggest appropriate grasp types.
• Tactile feedback is collected via a forearm-mounted touchscreen, allowing users to correct grasp predictions in real time.
• Speech input from a wrist-mounted microphone enables natural language corrections, such as 'try a different grip'.
• A hybrid controller combines EMG-based predictions with visual and feedback inputs, using a neural network to update grasp models based on new training examples.
• The system performs offline retraining of the grasp classifier using newly acquired in-situ data to improve model accuracy over time.

Experimental results

Research questions

• RQ1 Can a prosthetic hand system learn and adapt grasp preferences through real-time multi-modal feedback from the user?
• RQ2 How does incorporating tactile and speech feedback improve grasp classification accuracy compared to EMG-only control?
• RQ3 To what extent can a system personalize grasp behavior for novel or unfamiliar objects using in-situ learning?
• RQ4 How does the addition of training examples from user feedback affect the model’s ability to generalize across different object types?
• RQ5 Can on-board multi-modal interaction enable a self-improving prosthetic control system without requiring pre-programmed mappings?

Key findings

• After initial training, the system correctly classified the grasp type for spherical (apple), cylindrical (cup), hook (pitcher), and lateral (gelatin box) objects with high accuracy.
• The dice, intended for pinch grasp, was only correctly classified 10% of the time initially, and the spoon (tripod grasp) was never correctly classified.
• After adding 20 training examples of the banana (a novel object), the model increased the probability of predicting the correct tripod grasp from 0% to 16%.
• After collecting and retraining on 120 images of the banana, the system achieved a 78% probability of correctly inferring the desired tripod grasp.
• The addition of new data improved grasp classification for previously trained objects: the spoon’s correct grasp probability rose from 0% to 45%, and the dice from 10% to 29%.
• The system demonstrated successful execution of all six grasp types (spherical, cylindrical, hook, lateral, pinch, tripod) on real objects, as shown in Figure 10(d).