[Paper Review] Deep Reinforcement Learning for Haptic Shared Control in Unknown Tasks

Franklin Cardeñoso Fernández, Wouter Caarls|arXiv (Cornell University)|Jan 15, 2021

Teleoperation and Haptic Systems4 citations

TL;DR

This paper proposes a task-agnostic haptic shared control system using deep reinforcement learning (DDPG) and a CNN (VGG16) to infer task intent from visual input, enabling personalized, adaptive assistance in teleoperation. The system learns user-specific policies that reduce task completion time and resistance by dynamically adjusting haptic feedback based on real-time user behavior and visual context.

ABSTRACT

Recent years have shown a growing interest in using haptic shared control (HSC) in teleoperated systems. In HSC, the application of virtual guiding forces decreases the user's control effort and improves execution time in various tasks, presenting a good alternative in comparison with direct teleoperation. HSC, despite demonstrating good performance, opens a new gap: how to design the guiding forces. For this reason, the challenge lies in developing controllers to provide the optimal guiding forces for the tasks that are being performed. This work addresses this challenge by designing a controller based on the deep deterministic policy gradient (DDPG) algorithm to provide the assistance, and a convolutional neural network (CNN) to perform the task detection, called TAHSC (Task Agnostic Haptic Shared Controller). The agent learns to minimize the time it takes the human to execute the desired task, while simultaneously minimizing their resistance to the provided feedback. This resistance thus provides the learning algorithm with information about which direction the human is trying to follow, in this case, the pick-and-place task. Diverse results demonstrate the successful application of the proposed approach by learning custom policies for each user who was asked to test the system. It exhibits stable convergence and aids the user in completing the task with the least amount of time possible.

Motivation & Objective

• To address the challenge of designing optimal guiding forces in haptic shared control (HSC) for unknown or dynamic tasks.
• To enable the controller to infer task intent implicitly from visual input (e.g., pick-and-place) without predefined task definitions.
• To personalize assistance by learning user-specific policies through reinforcement learning, adapting to individual user behavior.
• To improve task performance by minimizing both execution time and user resistance to haptic feedback.
• To develop a system that generalizes across users while maintaining stable convergence and adaptability in real-time teleoperation.

Proposed method

• A deep deterministic policy gradient (DDPG) algorithm is used to learn continuous control policies for haptic assistance in real-time teleoperation.
• A convolutional neural network (VGG16) processes visual input from the environment to infer task context and guide the RL policy.
• The reward function is designed to minimize task completion time and user resistance, with a threshold parameter tuned to user behavior.
• The system integrates a DDPG-based RL controller with a proportional teleoperation controller, allowing the agent to optimize the combined human-robot performance.
• User actions and visual observations are used as state inputs, with the RL agent learning from trial-and-error interactions in a simulated environment.
• The policy is trained end-to-end using on-policy interaction, where user behavior directly shapes the optimal assistance strategy.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning agent learn effective, user-specific haptic assistance policies without prior knowledge of the task?
• RQ2 How well can a CNN (VGG16) infer task intent from visual input in a teleoperation setting with unknown tasks?
• RQ3 To what extent does the learned policy reduce task completion time and user resistance compared to baseline teleoperation?
• RQ4 How does the system generalize across different users, and does user-specific fine-tuning improve performance?
• RQ5 Can the reward function effectively capture user intent when the threshold parameter is set based on one user’s behavior?

Key findings

• The system successfully learned user-specific policies through training, resulting in faster task completion times and reduced user resistance.
• Subject 2 showed a significant improvement in mean velocity and force reduction after training, indicating strong policy adaptation.
• Subject 1, despite low velocity and high time, demonstrated consistent policy learning, confirming the system's ability to adapt to diverse user behaviors.
• Subject 0 exhibited intermediate performance, confirming that the system generalizes across users but achieves optimal results only with user-specific training.
• The policy converged stably across all subjects, demonstrating robustness and reliability in real-time operation.
• The use of visual input (via VGG16) enabled dynamic task inference without explicit task definitions, supporting task-agnostic operation.