[Paper Review] Comparative Safety Performance of Autonomous- and Human Drivers: A Real-World Case Study of the Waymo One Service

Luigi Di Lillo, Tilia Gode+4 more|arXiv (Cornell University)|Sep 3, 2023

Traffic and Road Safety4 citations

TL;DR

This study evaluates the safety of Waymo One's autonomous vehicles by comparing real-world collision claims data against human-driven baselines. Using third-party insurance claims and mileage- and location-adjusted Swiss Re data, it finds that Waymo's driverless vehicles incurred zero bodily injury claims per million miles and 0.78 property damage claims per million miles—significantly lower than the human driver baseline of 1.11 and 3.26 cpmm, respectively, across over 3.8 million miles in rider-only mode.

ABSTRACT

This study compares the safety of autonomous- and human drivers. It finds that the Waymo One autonomous service is significantly safer towards other road users than human drivers are, as measured via collision causation. The result is determined by comparing Waymo's third party liability insurance claims data with mileage- and zip-code-calibrated Swiss Re (human driver) private passenger vehicle baselines. A liability claim is a request for compensation when someone is responsible for damage to property or injury to another person, typically following a collision. Liability claims reporting and their development is designed using insurance industry best practices to assess crash causation contribution and predict future crash contributions. In over 3.8 million miles driven without a human being behind the steering wheel in rider-only (RO) mode, the Waymo Driver incurred zero bodily injury claims in comparison with the human driver baseline of 1.11 claims per million miles (cpmm). The Waymo Driver also significantly reduced property damage claims to 0.78 cpmm in comparison with the human driver baseline of 3.26 cpmm. Similarly, in a more statistically robust dataset of over 35 million miles during autonomous testing operations (TO), the Waymo Driver, together with a human autonomous specialist behind the steering wheel monitoring the automation, also significantly reduced both bodily injury and property damage cpmm compared to the human driver baselines.

Motivation & Objective

• To assess the comparative safety performance of autonomous vehicles versus human drivers in real-world conditions.
• To evaluate whether Waymo One’s autonomous service reduces collision-related liability claims compared to human-driven vehicles.
• To establish a reliable benchmark for autonomous vehicle safety using third-party insurance claims data and calibrated human driver baselines.
• To analyze both rider-only (RO) and testing operations (TO) modes to ensure statistical robustness in safety comparisons.

Proposed method

• The study uses third-party liability insurance claims data from Waymo One’s operations as the primary safety metric.
• It calibrates human driver crash rates using Swiss Re private passenger vehicle baselines, adjusted for mileage and geographic location (zip code).
• Liability claims are used as a proxy for crash causation, with claims reporting and development following insurance industry best practices.
• The analysis compares Waymo’s claims per million miles (cpmm) for both bodily injury and property damage against the calibrated human driver baselines.
• Two operational modes are analyzed: rider-only (RO) mode with no human behind the wheel and testing operations (TO) mode with a human safety driver monitoring the system.
• Statistical robustness is enhanced by analyzing over 35 million miles in TO mode and over 3.8 million miles in RO mode.

Experimental results

Research questions

• RQ1 How does the frequency of liability claims for bodily injury compare between Waymo’s autonomous vehicles and human-driven vehicles?
• RQ2 What is the property damage claim rate for Waymo’s autonomous vehicles relative to human-driven vehicles?
• RQ3 Does the safety performance of autonomous vehicles remain significantly better when evaluated across multiple operational modes and large-scale real-world data?
• RQ4 To what extent do calibrated human driver baselines from Swiss Re support reliable comparison with autonomous vehicle performance?

Key findings

• In rider-only mode, Waymo Driver incurred zero bodily injury claims per million miles, compared to a human driver baseline of 1.11 claims per million miles.
• For property damage claims, Waymo Driver recorded 0.78 claims per million miles, significantly lower than the human driver baseline of 3.26 claims per million miles.
• In testing operations mode, which includes a human safety driver, Waymo Driver still significantly reduced both bodily injury and property damage claims compared to the human baseline.
• The safety advantage of autonomous vehicles was consistent across both RO and TO operational modes, indicating robust performance.
• The study’s findings are based on over 3.8 million miles driven in RO mode and over 35 million miles in TO mode, ensuring statistical reliability.
• The use of insurance claims data calibrated to real-world human driver baselines provides a credible, industry-standard method for assessing real-world safety performance.