[Paper Review] Shared Control Based on Extended Lipschitz Analysis With Application to Human-Superlimb Collaboration

Hanjun Song, H. Harry Asada|arXiv (Cornell University)|Sep 1, 2023

Stroke Rehabilitation and Recovery4 citations

TL;DR

This paper proposes a data-driven shared control framework for human-superlimb collaboration that uses extended Lipschitz analysis to quantitatively assign tasks between human voluntary control and robot reactive control. By identifying predictable motion patterns via low Lipschitz quotients and augmenting input space with task-mode information from Hidden Markov Models, the method assigns predictable actions to the robot and unpredictable, spontaneous motions (e.g., reciprocating knife cuts) to the human, achieving effective autonomy-intervention trade-offs in bimanual eating tasks with a Supernumerary Robotic Limb.

ABSTRACT

This paper presents a quantitative method to construct voluntary manual control and sensor-based reactive control in human-robot collaboration based on Lipschitz conditions. To collaborate with a human, the robot observes the human's motions and predicts a desired action. This predictor is constructed from data of human demonstrations observed through the robot's sensors. Analysis of demonstration data based on Lipschitz quotients evaluates a) whether the desired action is predictable and b) to what extent the action is predictable. If the quotients are low for all the input-output pairs of demonstration data, a predictor can be constructed with a smooth function. In dealing with human demonstration data, however, the Lipschitz quotients tend to be very high in some situations due to the discrepancy between the information that humans use and the one robots can obtain. This paper a) presents a method for seeking missing information or a new variable that can lower the Lipschitz quotients by adding the new variable to the input space, and b) constructs a human-robot shared control system based on the Lipschitz analysis. Those predictable situations are assigned to the robot's reactive control, while human voluntary control is assigned to those situations where the Lipschitz quotients are high even after the new variable is added. The latter situations are deemed unpredictable and are rendered to the human. This human-robot shared control method is applied to assist hemiplegic patients in a bimanual eating task with a Supernumerary Robotic Limb, which works in concert with an unaffected functional hand.

Motivation & Objective

• To develop a quantitative method for determining the appropriate division of labor between human and robot in human-robot collaboration.
• To address the challenge of unpredictable human motions in robotic control by identifying confounding input-output relationships using Lipschitz conditions.
• To improve predictability of human demonstrations by discovering missing variables or hidden states that reduce Lipschitz quotients.
• To integrate voluntary and reactive control in a shared control framework using a weighted sum based on predictability, minimizing human workload and robot interference.
• To validate the method in a real-world application: assisting hemiplegic patients with bimanual eating using a Supernumerary Robotic Limb.

Proposed method

• Lipschitz quotients are computed from human demonstration data to assess the predictability of input-output mappings, with high quotients indicating non-predictable or confounded relationships.
• A Hidden Markov Model (HMM) is used to infer task modes from demonstration data, which are then added to the input space to resolve confounding cases and reduce Lipschitz quotients.
• The method identifies situations where Lipschitz quotients remain high even after adding task-mode information, designating them as unpredictable and assigning them to human voluntary control.
• Reactive control is applied to data with low Lipschitz quotients, while voluntary control is assigned to high-quotient cases, with a weighted sum of controls used for smooth integration.
• The framework uses a threshold on Lipschitz quotients to tune the trade-off between robot autonomy and human intervention.
• The approach is validated in a bimanual eating task where a SuperLimb assists a hemiplegic patient by holding a knife, coordinating with the patient’s functional hand.

Experimental results

Research questions

• RQ1 How can Lipschitz conditions be extended to quantify the predictability of human motion in human-robot collaboration?
• RQ2 What missing or hidden variables can be identified to reduce Lipschitz quotients and improve the predictability of human demonstrations?
• RQ3 How can a shared control system be designed to assign predictable actions to robot reactive control and unpredictable actions to human voluntary control?
• RQ4 What is the trade-off between robot prediction accuracy and human control effort, and how can it be quantitatively managed using Lipschitz thresholds?
• RQ5 Can this method effectively support bimanual tasks in rehabilitation, such as eating with a SuperLimb, by minimizing interference and cognitive load?

Key findings

• Adding task-mode information from an HMM significantly reduced Lipschitz quotients for many motion patterns, improving the predictability of human demonstrations.
• Despite the addition of task-mode signals, certain motions—particularly reciprocating knife cuts—remained highly unpredictable due to spontaneous human control, justifying their assignment to voluntary control.
• The root mean square error (RMSE) of reactive control decreased as the Lipschitz constant threshold was lowered, indicating improved prediction accuracy.
• Voluntary control effort increased as the Lipschitz threshold was lowered, reflecting the trade-off between robot autonomy and human workload.
• The method successfully enabled a shared control system where robot reactive control did not interfere with human voluntary control in unpredictable situations.
• The framework demonstrated effective integration of human and robot control in a real-world bimanual eating task with a Supernumerary Robotic Limb, supporting hemiplegic patients.