[Paper Review] Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving

Bob Wei, Mengye Ren+4 more|arXiv (Cornell University)|Nov 2, 2020

Advanced Neural Network Applications54 references4 citations

TL;DR

This paper proposes a learnable, sparse spatial attention module in an end-to-end self-driving network that directly optimizes for motion planning safety, not perception accuracy. By focusing computation on critical scene regions via a data-driven attention mask, the model reduces collision rates by 20% and improves planning performance at 95% sparsity, while enhancing interpretability through attention visualization.

ABSTRACT

In this paper, we propose an end-to-end self-driving network featuring a sparse attention module that learns to automatically attend to important regions of the input. The attention module specifically targets motion planning, whereas prior literature only applied attention in perception tasks. Learning an attention mask directly targeted for motion planning significantly improves the planner safety by performing more focused computation. Furthermore, visualizing the attention improves interpretability of end-to-end self-driving.

Motivation & Objective

• Address the misalignment between perception objectives (e.g., average precision) and the true goal of safe motion planning in end-to-end self-driving systems.
• Reduce computational waste on irrelevant or distant objects that do not impact driving safety.
• Improve motion planning safety and efficiency by learning an attention mask directly optimized for planning performance.
• Enhance interpretability of end-to-end models through visualized attention masks that highlight relevant scene regions.

Proposed method

• Introduce a sparse attention module that gates feature backbone computation, selectively focusing on spatial regions critical for motion planning.
• Train the attention mask end-to-end using a multi-task loss that reweights detection and forecasting losses based on attention mask regions.
• Apply an ℓ₁ regularization term to encourage sparsity in the attention mask, with sparsity controlled by a hyperparameter λₐ.
• Use a learnable loss weighting ratio γ₁ to balance perception and prediction (PnP) losses between actors inside and outside the attention mask.
• Optimize the attention mask using a joint training objective that prioritizes planning performance over full-scene detection accuracy.
• Visualize the attention mask to improve interpretability and validate that the model focuses on relevant traffic participants and road regions.

Experimental results

Research questions

• RQ1 Can a learnable spatial attention mechanism improve motion planning safety in end-to-end self-driving systems by focusing computation on relevant scene regions?
• RQ2 Does learning an attention mask directly for the motion planning objective lead to better performance and efficiency than standard perception-focused training?
• RQ3 How does sparsity in the attention mask affect planning performance and computational efficiency?
• RQ4 Can attention-based loss reweighting improve planning performance without degrading detection accuracy on critical actors?
• RQ5 To what extent does visualizing the attention mask enhance interpretability of end-to-end self-driving models?

Key findings

• The proposed model achieves a 20% reduction in collision rate (0.511% vs. 0.639%) at 95% attention sparsity compared to the dense baseline.
• Planning performance improves with increased sparsity, peaking at 95% sparsity with a 3-second trajectory L2 error of 2.102m.
• The model outperforms the dense baseline in mAP on the attended region (88.5% IoU@0.7) despite being 95% sparse, indicating better focus on relevant actors.
• Loss reweighting with γ₁ = 0.9 yields optimal performance, while γ₁ = 1.0 leads to poor performance due to overfitting to attention mask regions.
• The attention mask visualization shows that the model learns to focus on relevant traffic participants and road areas, improving interpretability.
• Failure cases are primarily due to rear-end collisions, suggesting limited focus on open road options like right turns in open-loop evaluation.