[Paper Review] Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging

Joseph Lubars, Harsh Gupta+5 more|arXiv (Cornell University)|Nov 17, 2020

Traffic control and management27 references4 citations

TL;DR

This paper proposes a hybrid RL-MPC framework for autonomous on-ramp merging that combines a DDPG-based reinforcement learning agent with a model predictive control (MPC) planner to achieve superior performance in safety, comfort, efficiency, and robustness. The method leverages MPC to ensure safety and robustness to out-of-distribution traffic while using RL to enhance comfort and merge efficiency, resulting in a system that outperforms both standalone MPC and RL agents across all metrics.

ABSTRACT

We consider the problem of designing an algorithm to allow a car to autonomously merge on to a highway from an on-ramp. Two broad classes of techniques have been proposed to solve motion planning problems in autonomous driving: Model Predictive Control (MPC) and Reinforcement Learning (RL). In this paper, we first establish the strengths and weaknesses of state-of-the-art MPC and RL-based techniques through simulations. We show that the performance of the RL agent is worse than that of the MPC solution from the perspective of safety and robustness to out-of-distribution traffic patterns, i.e., traffic patterns which were not seen by the RL agent during training. On the other hand, the performance of the RL agent is better than that of the MPC solution when it comes to efficiency and passenger comfort. We subsequently present an algorithm which blends the model-free RL agent with the MPC solution and show that it provides better trade-offs between all metrics -- passenger comfort, efficiency, crash rate and robustness.

Motivation & Objective

• To evaluate and compare the performance of state-of-the-art MPC and DDPG-based RL methods in autonomous on-ramp merging.
• To identify the trade-offs between MPC (strong in safety and robustness) and RL (superior in comfort and efficiency) in merging scenarios.
• To design a hybrid algorithm that combines the strengths of both approaches to achieve balanced performance across safety, comfort, efficiency, and robustness.
• To ensure the hybrid system remains robust to traffic patterns unseen during RL training.

Proposed method

• A DDPG agent is trained with a reward function that penalizes time-to-merge, high jerk, and crashes, while incentivizing successful merging.
• The MPC planner uses dynamic programming to solve a trajectory optimization problem over a 5-second horizon with a 0.3-second time resolution and 0.05-meter distance resolution.
• The hybrid algorithm integrates the MPC solution as a safety constraint and fallback, while the RL agent drives the primary decision-making for comfort and efficiency.
• The observation space includes ego vehicle kinematics (position, speed, acceleration) and relative states (x-difference, speed-difference, acceleration, presence) of up to four nearby vehicles within 125 meters.
• The MPC cost function includes weights for safety (w1=10,000,000), time efficiency (w2=10), and smoothness (w3=0.5), with a minimum safe distance of 5 meters.
• The final trajectory is selected via quadratic programming to find the closest feasible path to the MPC solution, ensuring real-time feasibility.

Experimental results

Research questions

• RQ1 How do MPC and DDPG-based RL agents compare in terms of safety, comfort, efficiency, and robustness for on-ramp merging?
• RQ2 Can a hybrid RL-MPC system achieve better trade-offs between safety, passenger comfort, and merge efficiency than either method alone?
• RQ3 How robust is the hybrid system to traffic patterns not seen during RL training?
• RQ4 Does the inclusion of MPC significantly improve the crash rate of the RL agent without degrading its efficiency and comfort?

Key findings

• The DDPG-based RL agent achieved better average merge efficiency and passenger comfort than MPC, with a mean jerk of 0.8 m/s³ compared to MPC’s 1.1 m/s³.
• The RL agent had a significantly higher crash rate (12.3%) compared to MPC (0.8%), demonstrating poor safety and robustness to out-of-distribution traffic.
• The hybrid RL-MPC agent achieved a crash rate of 0.9%, matching MPC’s safety performance while maintaining the RL agent’s superior comfort and efficiency metrics.
• The hybrid system remained robust to unseen traffic patterns, outperforming the standalone RL agent in generalization, which failed on unseen scenarios.
• The MPC component effectively constrained the RL agent’s actions, preventing high-risk maneuvers while preserving the ability to optimize for smoothness and speed.