[Paper Review] PlanT: Explainable Planning Transformers via Object-Level Representations

Katrin Renz, Kashyap Chitta+4 more|arXiv (Cornell University)|Oct 25, 2022

Autonomous Vehicle Technology and Safety21 citations

TL;DR

PlanT introduces a transformer-based planner using compact object-level inputs (vehicles and route) for autonomous driving, achieving expert-level performance with fast inference and enhanced explainability via attention. It also enables sensor-based planning when paired with a perception module.

ABSTRACT

Planning an optimal route in a complex environment requires efficient reasoning about the surrounding scene. While human drivers prioritize important objects and ignore details not relevant to the decision, learning-based planners typically extract features from dense, high-dimensional grid representations containing all vehicle and road context information. In this paper, we propose PlanT, a novel approach for planning in the context of self-driving that uses a standard transformer architecture. PlanT is based on imitation learning with a compact object-level input representation. On the Longest6 benchmark for CARLA, PlanT outperforms all prior methods (matching the driving score of the expert) while being 5.3x faster than equivalent pixel-based planning baselines during inference. Combining PlanT with an off-the-shelf perception module provides a sensor-based driving system that is more than 10 points better in terms of driving score than the existing state of the art. Furthermore, we propose an evaluation protocol to quantify the ability of planners to identify relevant objects, providing insights regarding their decision-making. Our results indicate that PlanT can focus on the most relevant object in the scene, even when this object is geometrically distant.

Motivation & Objective

• Motivate planning for autonomous driving using compact object-level representations rather than dense pixel-level inputs.
• Develop a transformer-based planner that operates on vehicles and route segments to predict ego-trajectory.
• Show that object-level planning can achieve state-of-the-art performance with efficient inference.
• Propose an explainability protocol to identify which objects most influence planning decisions.

Proposed method

• Represent a scene as a set of object tokens (vehicles and route segments) with 6 attributes each and embed them for a transformer encoder.
• Use a [CLS]-token based transformer encoder to aggregate scene information and generate future waypoints via a GRU-based decoder.
• Add a binary traffic light flag to initialize the decoder and predict a 4-step BEV trajectory.
• Introduce an auxiliary task predicting future attributes of other vehicles by discretizing attributes into bins and applying cross-entropy losses.
• Train with a multi-task loss combining waypoint L1 loss and vehicle attribute cross-entropy losses (λ-weighted).

Experimental results

Research questions

• RQ1 Can an object-level, transformer-based representation match or exceed pixel-based planners on urban driving benchmarks?
• RQ2 What input components (360-degree view, speeds, route representation) are critical for planning performance?
• RQ3 How explainable is PlanT, and can attention identify the most relevant objects for safe planning?
• RQ4 Does PlanT transfer to sensor-based planning when combined with perception modules?
• RQ5 What is the impact of scaling data and model size on planning performance?

Key findings

• PlanT attains expert-level driving performance on Longest6 with faster inference than pixel-based baselines.
• A full 360-degree field of view, vehicle speeds, and larger data/model sizes significantly improve PlanT and PlanCNN performance.
• PlanT with perception outperforms TransFuser and LAV by large margins (DS improvements cited in Table 1).
• PlanT’s attention weights provide reliable explanations of object relevance, outperforming PlanCNN in RFDS.
• PlanT achieves 81.36 DS on the 3× dataset (MEDIUM) and is ~3× faster than PlanCNN; PlanT MINI reaches 5.3× faster with similar DS.