[Paper Review] Query-Efficient Imitation Learning for End-to-End Autonomous Driving

Jiakai Zhang, Kyunghyun Cho|arXiv (Cornell University)|May 20, 2016

Reinforcement Learning in Robotics15 references115 citations

TL;DR

SafeDAgger extends DAgger with a safety policy to reduce reference-policy queries, enabling more query-efficient imitation learning for end-to-end autonomous driving and faster, safer convergence in TORCS simulations.

ABSTRACT

One way to approach end-to-end autonomous driving is to learn a policy function that maps from a sensory input, such as an image frame from a front-facing camera, to a driving action, by imitating an expert driver, or a reference policy. This can be done by supervised learning, where a policy function is tuned to minimize the difference between the predicted and ground-truth actions. A policy function trained in this way however is known to suffer from unexpected behaviours due to the mismatch between the states reachable by the reference policy and trained policy functions. More advanced algorithms for imitation learning, such as DAgger, addresses this issue by iteratively collecting training examples from both reference and trained policies. These algorithms often requires a large number of queries to a reference policy, which is undesirable as the reference policy is often expensive. In this paper, we propose an extension of the DAgger, called SafeDAgger, that is query-efficient and more suitable for end-to-end autonomous driving. We evaluate the proposed SafeDAgger in a car racing simulator and show that it indeed requires less queries to a reference policy. We observe a significant speed up in convergence, which we conjecture to be due to the effect of automated curriculum learning.

Motivation & Objective

• Motivate end-to-end autonomous driving via imitation learning from a reference policy.
• Address the high query cost of DAgger when the reference policy is expensive (e.g., human drivers).
• Propose SafeDAgger, a query-efficient extension of DAgger with a safety policy to minimize reference-policy queries.
• Demonstrate through TORCS simulations that SafeDAgger speeds up convergence and reduces crashes/damage.
• Highlight the automated curriculum learning effect due to subset selection guided by safety assessments.

Proposed method

• Introduce a safety policy that predicts when the primary policy may deviate from the reference policy without querying it.
• Define deviation ε(π,π*,φ(s)) = ||π(φ(s)) − π*(φ(s))||^2 and a threshold τ to form π_safe*.
• Integrate the safety policy into a SafeDAgger loop that collects only hard examples (where safety policy returns 0) for querying the reference policy.
• Use subset selection to limit queried states during data collection, enabling data efficiency and curriculum-like learning.
• Maintain a learning-to-search framework similar to DAgger, with updates to both the primary policy and the safety policy across iterations.
• Apply to TORCS with a deep CNN primary policy predicting steering, brake, and affordances, plus a safety policy predicting safe/unsafe drive decisions.

Experimental results

Research questions

• RQ1 Does SafeDAgger reduce the number of queries to the reference policy compared to standard DAgger in end-to-end driving?
• RQ2 Can SafeDAgger achieve faster convergence and better driving performance (fewer crashes, lower damage) than supervised learning or DAgger in a simulated driving environment?
• RQ3 Does the safety policy enable a meaningful automated curriculum that improves data efficiency and policy quality?
• RQ4 How does SafeDAgger perform under traffic versus no-traffic conditions in TORCS?
• RQ5 Is it feasible to generalize the safety-policy concept to other imitation-learning frameworks beyond DAgger?

Key findings

• SafeDAgger requires substantially fewer queries to the reference policy than original DAgger during training.
• After three iterations, SafeDAgger-trained policies achieve near-perfect driving in the TORCS setup.
• The safety policy reduced the time the reference policy was used during test, with 7.11% without traffic and 10.81% with traffic, at early stages.
• Approximately 77.70% of training examples are deemed safe by the safety policy in the reported setup.
• SafeDAgger exhibits faster convergence and a clearer downward trend in the reliance on the reference policy compared to vanilla DAgger.