[Paper Review] Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF

Sun Yuan, Navid Salami Pargoo+2 more|arXiv (Cornell University)|Jun 6, 2024

Human-Automation Interaction and SafetyPsychology3 citations

TL;DR

This paper proposes a human-centric, LLM-enhanced Reinforcement Learning from Human Feedback (RLHF) framework for autonomous driving safety. By integrating physical and physiological sensor data from human drivers and using LLMs to interpret and simulate human preferences, the method fine-tunes autonomous vehicle policies in a multi-agent simulation, significantly improving alignment with real-world human behavior and safety standards.

ABSTRACT

Reinforcement Learning from Human Feedback (RLHF) is popular in large language models (LLMs), whereas traditional Reinforcement Learning (RL) often falls short. Current autonomous driving methods typically utilize either human feedback in machine learning, including RL, or LLMs. Most feedback guides the car agent's learning process (e.g., controlling the car). RLHF is usually applied in the fine-tuning step, requiring direct human "preferences," which are not commonly used in optimizing autonomous driving models. In this research, we innovatively combine RLHF and LLMs to enhance autonomous driving safety. Training a model with human guidance from scratch is inefficient. Our framework starts with a pre-trained autonomous car agent model and implements multiple human-controlled agents, such as cars and pedestrians, to simulate real-life road environments. The autonomous car model is not directly controlled by humans. We integrate both physical and physiological feedback to fine-tune the model, optimizing this process using LLMs. This multi-agent interactive environment ensures safe, realistic interactions before real-world application. Finally, we will validate our model using data gathered from real-life testbeds located in New Jersey and New York City.

Motivation & Objective

• To address the gap in applying Reinforcement Learning from Human Feedback (RLHF) to autonomous driving, where frame-by-frame human preference feedback is impractical.
• To enhance safety and realism in autonomous vehicle training by simulating human-driven agents and pedestrians in a multi-agent environment.
• To integrate physical (e.g., steering, braking) and physiological (e.g., heart rate, gaze) feedback from human drivers to model human preferences.
• To leverage LLMs to interpret multimodal human feedback and generate synthetic human behaviors for scalable training.
• To validate the framework using real-world data from testbeds in New Jersey and New York City, enabling simulation-to-reality transfer.

Proposed method

• Train an autonomous vehicle agent using a pre-trained policy, fine-tuned via RLHF using human feedback collected during simulation.
• Integrate human drivers using physical controllers (steering wheel, pedals) and wearable sensors (Empatica for physiological signals, Adhawk for gaze tracking).
• Use a VR headset and Raspberry Pi camera to capture immersive environment interactions and real-time human reactions.
• Employ an LLM agent (e.g., GPT-4) to interpret multimodal sensor data and convert it into actionable preference signals for the RLHF loop.
• Simulate multiple human agents (drivers and pedestrians) via LLM-driven behavior cloning to enrich interaction diversity and realism.
• Import real-world road data from New York City and New Jersey testbeds into the CARLA simulation for cross-validation and real-world alignment.

Figure 1. Overview of our human-centric multi-agent LLM-enhanced RLHF system framework. During the fine-tuning of an autonomous car model, human agents and proliferated LLM agents mimicking multiple human behaviors are incorporated into the environment to align with real-world human preferences.

Experimental results

Research questions

• RQ1 Can RLHF be effectively applied to autonomous driving by modeling human preferences through multimodal sensor feedback instead of direct preference labeling?
• RQ2 How does incorporating both physical and physiological feedback improve the safety and realism of autonomous vehicle decision-making?
• RQ3 To what extent can LLMs simulate diverse human driving behaviors to enhance training in multi-agent simulation environments?
• RQ4 How does the integration of real-world testbed data into simulation improve the robustness and generalization of the trained autonomous driving policy?
• RQ5 What impact do varying human driving skill levels have on the performance and learning efficiency of the RLHF framework?

Key findings

• The framework successfully integrates physical and physiological feedback from human drivers into the RLHF loop, enabling more human-aligned policy learning.
• LLM agents effectively interpret multimodal human data and simulate realistic human behaviors, including overtaking and collision avoidance, enhancing training diversity.
• Initial implementation using GPT-4 demonstrates functional LLM-driven behavior imitation and guidance in CARLA simulations, including collision avoidance and user assistance.
• Real-world road data from New York City and New Jersey testbeds has been successfully imported into the CARLA simulation for validation and cross-domain testing.
• The system enables scalable simulation of human agents through LLM proliferation, reducing reliance on real human participants during training.
• Preliminary results confirm the theoretical feasibility and infrastructure readiness of the proposed simulation-to-reality framework for safe autonomous driving.

Figure 2. Simulation room with VR headset, steering controls, and monitors for real-time multimodal data collection and autonomous driving optimization.