[Paper Review] Reinforcement Learning-based Virtual Fixtures for Teleoperation of Hydraulic Construction Machine

Hyung Joo Lee, Sigrid Brell‐Çokcan|arXiv (Cornell University)|Jun 20, 2023

BIM and Construction Integration4 citations

TL;DR

This paper proposes a reinforcement learning (RL)-based virtual fixture system to enhance teleoperation of hydraulic construction machines by learning optimal joint coordination policies from data-driven simulations. The framework reduces operator mental workload and improves task efficiency by guiding joystick inputs through learned virtual fixtures, demonstrated in a user study showing reduced effort and smoother end-effector paths during chisel insertion into a borehole.

ABSTRACT

The utilization of teleoperation is a crucial aspect of the construction industry, as it enables operators to control machines safely from a distance. However, remote operation of these machines at a joint level using individual joysticks necessitates extensive training for operators to achieve proficiency due to their multiple degrees of freedom. Additionally, verifying the machine resulting motion is only possible after execution, making optimal control challenging. In addressing this issue, this study proposes a reinforcement learning-based approach to optimize task performance. The control policy acquired through learning is used to provide instructions on efficiently controlling and coordinating multiple joints. To evaluate the effectiveness of the proposed framework, a user study is conducted with a Brokk 170 construction machine by assessing its performance in a typical construction task involving inserting a chisel into a borehole. The effectiveness of the proposed framework is evaluated by comparing the performance of participants in the presence and absence of virtual fixtures. This study results demonstrate the proposed framework potential in enhancing the teleoperation process in the construction industry.

Motivation & Objective

• To address the high cognitive and motor demands of teleoperating multi-degree-of-freedom construction machines with individual joysticks.
• To reduce operator error and improve task efficiency in unstructured construction environments where real-time feedback is limited.
• To develop a data-driven, simulation-based RL framework that captures machine-specific nonlinear dynamics without requiring detailed analytical models.
• To evaluate the effectiveness of virtual fixtures derived from trained RL policies in a real-world teleoperation task.
• To minimize the simulation-to-reality gap by integrating a data-driven actuator model into the training environment.

Proposed method

• A data-driven actuator model is trained using real machine operation data to accurately simulate nonlinear hydraulic dynamics in simulation.
• A deep reinforcement learning agent is trained in a parallelized simulation environment with 128 agents to learn the hammer insertion task.
• The trained RL policy generates optimal joint trajectories and is used to define virtual fixtures that guide human operators.
• Virtual fixtures are implemented as visual and kinesthetic constraints that suggest optimal joystick movements and joint coordination patterns.
• The system uses a dynamic simulator that integrates the data-driven actuator model to ensure realistic motion during training and evaluation.
• A user study compares teleoperation performance with and without virtual fixtures using a Brokk 170 machine in a borehole insertion task.

Experimental results

Research questions

• RQ1 Can a data-driven RL framework learn efficient joint coordination policies for teleoperation of hydraulic construction machines with minimal simulation-to-reality gap?
• RQ2 How effective are virtual fixtures derived from trained RL policies in improving operator performance during complex teleoperation tasks?
• RQ3 To what extent do virtual fixtures reduce mental workload and physical effort in teleoperation of multi-jointed construction machines?
• RQ4 Can virtual fixtures guide operators to use multiple joints simultaneously, leading to smoother and more efficient end-effector paths?
• RQ5 How does the integration of real machine data into the RL training process improve policy generalization and task performance?

Key findings

• Participants using virtual fixtures demonstrated significantly reduced mental workload and perceived effort, as confirmed by NASA-TLX questionnaire results.
• The use of virtual fixtures led to more coordinated joint control, with operators tending to move multiple joints simultaneously rather than sequentially.
• End-effector paths with virtual fixtures were smoother and required fewer movements, resulting in faster and more accurate insertion into the borehole.
• The virtual fixtures guided operators to maintain optimal distance to the borehole, reducing the number of corrective motions needed.
• The data-driven actuator model effectively reduced the simulation-to-reality gap, enabling the trained policy to generalize well to real-world teleoperation.
• The framework successfully enabled safe and efficient teleoperation of a complex hydraulic construction machine in a dynamic, unstructured environment.