[Paper Review] Sensor Fusion for Predictive Control of Human-Prosthesis-Environment Dynamics in Assistive Walking: A Survey

Kuangen Zhang, Clarence W. de Silva+1 more|arXiv (Cornell University)|Mar 18, 2019

Prosthetics and Rehabilitation RoboticsEngineering59 references22 citations

TL;DR

This survey proposes a sensor fusion framework for predictive control of human-prosthesis-environment dynamics in assistive walking, integrating multimodal sensors (IMU, EMG, vision) to enhance intent recognition and environmental awareness. By fusing stationary and deep features with online learning, it enables robust, adaptive control for powered lower limb prostheses in complex environments.

ABSTRACT

This survey paper concerns Sensor Fusion for Predictive Control of Human-Prosthesis-Environment Dynamics in Assistive Walking. The powered lower limb prosthesis can imitate the human limb motion and help amputees to recover the walking ability, but it is still a challenge for amputees to walk in complex environments with the powered prosthesis. Previous researchers mainly focused on the interaction between a human and the prosthesis without considering the environmental information, which can provide an environmental context for human-prosthesis interaction. Therefore, in this review, recent sensor fusion methods for the predictive control of human-prosthesis-environment dynamics in assistive walking are critically surveyed. In that backdrop, several pertinent research issues that need further investigation are presented. In particular, general controllers, comparison of sensors, and complete procedures of sensor fusion methods that are applicable in assistive walking are introduced. Also, possible sensor fusion research for human-prosthesis-environment dynamics is presented.

Motivation & Objective

• Address the limitation of existing prosthetic control that ignores environmental context by integrating human-prosthesis-environment dynamics.
• Overcome user-dependent and non-stationary sensor signals (e.g., EMG, EMG) through robust sensor fusion for reliable intent prediction.
• Enable predictive walking control by fusing environmental perception with human motion intent recognition to reconstruct the broken vision-locomotion loop.
• Improve long-term performance of high-level prosthetic controllers through online dataset updating and adaptive training.
• Advance the integration of classification and regression in sensor fusion to enhance joint torque estimation and control robustness.

Proposed method

• Employ a hierarchical control architecture with high-level, mid-level, and low-level controllers, focusing on sensor fusion for the high-level controller.
• Integrate multiple sensors—IMUs for kinematics, EMG for neuromuscular intent, and vision for environmental context—using fusion techniques.
• Extract stationary features from signals (e.g., kinematic models, muscle-tendon dynamics) to improve generalization across users and sessions.
• Utilize deep neural networks to learn deep features automatically from raw sensor data, with constrained complexity for real-time use.
• Implement online dataset updating using delayed human activity recognition (via IMU) to label and retrain models for non-stationary signals.
• Combine classification (motion primitives) and regression (joint torque) outputs through sensor fusion to improve control accuracy and smoothness.

Experimental results

Research questions

• RQ1 How can sensor fusion improve the prediction of human walking intent in powered lower limb prostheses when individual sensors are limited by noise and user dependency?
• RQ2 What role does environmental context play in enhancing the accuracy and robustness of prosthetic control in complex walking environments?
• RQ3 How can stationary or deep features be extracted from non-stationary wearable signals (e.g., EMG, MMG) to improve generalization across users and over time?
• RQ4 What is the impact of online dataset updating and retraining on the long-term performance of intent recognition models in prosthetic control?
• RQ5 How can classification of motion primitives and regression of joint torques be jointly optimized through sensor fusion for better control outcomes?

Key findings

• Sensor fusion of IMU, EMG, and vision data significantly improves the robustness and accuracy of human motion intent recognition compared to single-sensor approaches.
• Stationary features derived from kinematic and muscle-tendon dynamics models enhance the generalization of high-level controllers across different users and sessions.
• Online dataset updating using delayed IMU-based activity recognition enables continuous retraining, maintaining high accuracy for non-stationary signals like EMG.
• Combining classification of motion primitives with regression of joint torques through sensor fusion leads to more stable and predictive control than direct proportional control.
• Deep neural networks can effectively learn discriminative features from raw sensor data, but require architectural constraints for real-time applicability in prosthetic systems.
• The integration of environmental perception with human intent recognition reconstructs the vision-locomotion loop, enabling adaptive navigation in complex environments.