[Paper Review] Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles

Ionel Gog, Sukrit Kalra+4 more|arXiv (Cornell University)|Apr 15, 2021

Autonomous Vehicle Technology and Safety58 references62 citations

TL;DR

Pylot is an open-source, modular platform for autonomous vehicle (AV) research that enables systematic study of latency-accuracy tradeoffs in AV perception and planning pipelines. Built on a high-performance dataflow architecture, it integrates with simulators like CARLA and real vehicles, providing reference and ground-truth implementations to evaluate component performance under realistic end-to-end driving conditions, achieving top results in the CARLA Autonomous Driving Challenge.

ABSTRACT

We present Pylot, a platform for autonomous vehicle (AV) research and development, built with the goal to allow researchers to study the effects of the latency and accuracy of their models and algorithms on the end-to-end driving behavior of an AV. This is achieved through a modular structure enabled by our high-performance dataflow system that represents AV software pipeline components (object detectors, motion planners, etc.) as a dataflow graph of operators which communicate on data streams using timestamped messages. Pylot readily interfaces with popular AV simulators like CARLA, and is easily deployable to real-world vehicles with minimal code changes. To reduce the burden of developing an entire pipeline for evaluating a single component, Pylot provides several state-of-the-art reference implementations for the various components of an AV pipeline. Using these reference implementations, a Pylot-based AV pipeline is able to drive a real vehicle, and attains a high score on the CARLA Autonomous Driving Challenge. We also present several case studies enabled by Pylot, including evidence of a need for context-dependent components, and per-component time allocation. Pylot is open source, with the code available at https://github.com/erdos-project/pylot.

Motivation & Objective

• To address the lack of end-to-end evaluation frameworks that account for both latency and accuracy in AV components.
• To enable researchers to study how component runtime and accuracy jointly affect overall driving behavior and safety.
• To provide a portable, modular platform that supports seamless transition between simulation and real-world deployment.
• To improve debuggability and reproducibility of AV experiments through deterministic execution and dataflow-based architecture.
• To support A/B testing of AV components under realistic driving scenarios, including emergency maneuvers and high-speed conditions.

Proposed method

• Models and algorithms are structured as a dataflow graph of timestamped operators, enabling modular, composable AV pipeline construction.
• The platform uses a high-performance, deterministic dataflow system to ensure reproducible execution and low-latency communication between components.
• It integrates with simulation environments like CARLA and supports deployment to real vehicles with minimal code changes.
• Reference implementations are provided for perception (e.g., object detection, semantic segmentation), prediction, planning, and control modules.
• Ground-truth implementations allow for idealized testing and debugging by accessing simulator-provided state data.
• Pylot enables 'timely' evaluation metrics (e.g., timely mIoU, timely AP50) that account for model runtime and driving speed.

Experimental results

Research questions

• RQ1 How does the runtime of perception components affect their timely accuracy under different driving speeds?
• RQ2 What is the impact of component latency on end-to-end driving behavior, particularly during emergency maneuvers?
• RQ3 How do different planning algorithm configurations trade off between collision avoidance and ride comfort?
• RQ4 Which object detectors achieve optimal timely performance at various driving speeds and runtimes?
• RQ5 To what extent do context-dependent factors such as speed and occlusion influence the effectiveness of AV components?

Key findings

• A 10ms runtime for a semantic segmentation model reduces timely mIoU to approximately 0.75, which is below the mIoU of top Cityscapes challenge submissions.
• For pedestrian detection, timely AP50 halves when model runtime reaches 35ms, indicating a strong need for fast inference or trajectory prediction to compensate.
• At 40m/s, timely mIoU for semantic segmentation drops by 28% compared to 10m/s, and timely AP50 for detection drops to 0, demonstrating a severe degradation in accuracy at high speeds.
• Fast, low-accuracy detectors outperform slow, high-accuracy detectors in terms of timely accuracy when driving at high speeds, highlighting the importance of runtime in model selection.
• FOT planner with 30ms P99 runtime achieved better ride comfort (lower lateral jerk) than RRT* and Hybrid A*, despite fewer successful collision avoidance cases, illustrating a tradeoff between safety and comfort.
• The platform enabled a high-scoring submission in the CARLA Autonomous Driving Challenge HD map track, validating its effectiveness in real-world benchmarking.