[Paper Review] TARGET: Automated Scenario Generation from Traffic Rules for Testing Autonomous Vehicles via Validated LLM-Guided Knowledge Extraction

Yao Deng|arXiv (Cornell University)|May 10, 2023

Software Testing and Debugging Techniques4 citations

TL;DR

TARGET is an end-to-end framework that automates the generation of autonomous driving test scenarios from natural language traffic rules using a large language model (GPT-4) to extract and map rules into a custom domain-specific language (DSL). It produces executable scenario scripts for simulation, uncovering 700+ rule violations and system failures across four ADSs, with detailed logs enabling efficient debugging.

ABSTRACT

Recent incidents with autonomous vehicles highlight the need for rigorous testing to ensure safety and robustness. Constructing test scenarios for autonomous driving systems (ADSs), however, is labor-intensive. We propose TARGET, an end-to-end framework that automatically generates test scenarios from traffic rules. To address complexity, we leverage a Large Language Model (LLM) to extract knowledge from traffic rules. To mitigate hallucinations caused by large context during input processing, we introduce a domain-specific language (DSL) designed to be syntactically simple and compositional. This design allows the LLM to learn and generate test scenarios in a modular manner while enabling syntactic and semantic validation for each component. Based on these validated representations, TARGET synthesizes executable scripts to render scenarios in simulation. Evaluated seven ADSs with 284 scenarios derived from 54 traffic rules, TARGET uncovered 610 rule violations, collisions, and other issues. For each violation, TARGET generates scenario recordings and detailed logs, aiding root cause analysis. Two identified issues were confirmed by ADS developers: one linked to an existing bug report and the other to limited ADS functionality.

Motivation & Objective

• Address the labor-intensive and error-prone manual process of defining test scenarios for autonomous driving systems (ADSs) using natural language traffic rules.
• Overcome the limitations of existing DSLs and NLP-based rule extraction methods that require manual simplification or suffer from syntactic complexity.
• Enable automated, scalable, and accurate test scenario generation directly from real-world traffic rule descriptions in natural language.
• Improve the efficiency and effectiveness of ADS testing by reducing human effort and increasing coverage of edge cases and rule violations.
• Provide detailed diagnostic reports and scenario recordings to support root cause analysis and validation of detected issues in ADS behavior.

Proposed method

• Design a lightweight, expressive domain-specific language (DSL) tailored for describing traffic rule-based test scenarios with simplified syntax.
• Utilize GPT-4 via a multi-stage in-context few-shot learning prompt engineering approach to parse natural language traffic rules and map them to the DSL representation.
• Implement a template-based scenario generation pipeline that uses dictionary search and hierarchical map-based route search to populate scenario parameters accurately.
• Integrate with the Carla simulator using its native APIs to generate executable scenario scripts for simulation execution.
• Apply a rule-parsing pipeline that combines LLM inference with syntactic validation to ensure DSL compliance and improve parsing accuracy.
• Generate comprehensive test reports and scenario recordings for each test run, including logs of rule violations, collisions, and navigation failures.

Experimental results

Research questions

• RQ1 Can a large language model effectively extract and structure key elements from natural language traffic rules into a formal DSL representation for test scenario generation?
• RQ2 How effective is the proposed LLM-guided parsing pipeline in accurately converting real-world traffic rules into executable scenario scripts compared to manual or traditional NLP methods?
• RQ3 To what extent can the generated scenarios uncover previously unknown or existing rule violations and system failures in autonomous driving systems?
• RQ4 How does the framework’s automated scenario generation reduce the burden of manual test authoring while maintaining high scenario diversity and coverage?
• RQ5 Can the diagnostic outputs (logs and recordings) from the framework support reliable root cause analysis for identified ADS issues?

Key findings

• TARGET successfully generated 217 test scenarios across eight diverse maps using traffic rules from the Texas driver handbook.
• The framework identified approximately 700 distinct erroneous behaviors, including rule violations, collisions, and navigation failures, across four ADSs: LAV, MMFN, AUTO, and Autoware.
• Ablation studies confirmed that the proposed LLM-guided parsing pipeline outperformed direct GPT-4 generation alone, demonstrating the value of structured prompt engineering.
• The rule parser achieved 91.8% accuracy in representing benchmark traffic rules in the DSL, indicating strong understanding of natural language traffic rule semantics.
• Two previously unknown issues were confirmed by ADS developers: one related to perception failure (missed static objects in Autoware), and one to localization inaccuracy in MMFN.
• One of the detected issues was already documented in a public bug report, validating the framework’s ability to identify real-world ADS flaws.