[Paper Review] Gradient-free Policy Architecture Search and Adaptation

Sayna Ebrahimi, Anna Rohrbach+1 more|arXiv (Cornell University)|Oct 16, 2017

Reinforcement Learning in Robotics21 references12 citations

TL;DR

This paper proposes a gradient-free policy architecture search and adaptation method that jointly leverages expert demonstrations and environment rewards to learn safe, high-performing autonomous driving policies. By optimizing network architecture and adapting to domain shifts using only target-domain rewards, the approach reduces cumulative crashes and achieves superior performance compared to baseline methods.

ABSTRACT

We develop a method for policy architecture search and adaptation via gradient-free optimization which can learn to perform autonomous driving tasks. By learning from both demonstration and environmental reward we develop a model that can learn with relatively few early catastrophic failures. We first learn an architecture of appropriate complexity to perceive aspects of world state relevant to the expert demonstration, and then mitigate the effect of domain-shift during deployment by adapting a policy demonstrated in a source domain to rewards obtained in a target environment. We show that our approach allows safer learning than baseline methods, offering a reduced cumulative crash metric over the agent's lifetime as it learns to drive in a realistic simulated environment.

Motivation & Objective

• To address the challenge of unsafe exploration during policy training in autonomous driving by minimizing early catastrophic failures.
• To improve policy generalization across domain shifts (e.g., lighting, weather) without requiring target-domain demonstrations.
• To jointly optimize policy architecture and parameters using both expert demonstrations and environment rewards for safer, more efficient learning.
• To enable effective adaptation of a source-domain policy to a new, visually different target environment using only reward feedback.

Proposed method

• A gradient-free optimization algorithm inspired by [16], enhanced with improved noise generation for more accurate and efficient gradient estimation.
• Architecture search over variable-length neural network structures using evolutionary search to find optimal policy architectures for perception and control.
• Joint training of policy networks using behavioral cloning from expert demonstrations and reinforcement learning from environment rewards in the target domain.
• Adaptation of a pre-trained policy from a source domain to a target domain via reward-based fine-tuning, without requiring alignment or demonstrations in the target domain.
• Use of a composite reward function that balances accuracy and model size during architecture search.
• End-to-end training on the GTA5 simulation environment to evaluate performance on steering, braking, and throttle prediction.

Experimental results

Research questions

• RQ1 Can gradient-free architecture search improve policy performance and safety in autonomous driving compared to fixed or hand-designed architectures?
• RQ2 How effective is joint learning from demonstrations and environment rewards in reducing crashes during policy training?
• RQ3 To what extent can a policy learned in one domain be adapted to a new, visually different domain using only reward feedback?
• RQ4 Does optimizing policy architecture for both performance and compactness lead to better generalization and safety?

Key findings

• The proposed method reduced cumulative crashes during training compared to baselines relying solely on demonstrations or rewards.
• The adapted policy achieved 100% average reward after 53 hours of training (episode #832), outperforming a suboptimal baseline that plateaued at 97.3%.
• In complex scenes such as pedestrian crossings, the adapted model correctly predicted braking (0.956) and throttle (0.052), while the behaviorally cloned model incorrectly continued moving (brake: 0.191).
• The large network trained with both source and target rewards achieved a test loss of 3×10⁻⁵ on the target domain, significantly outperforming Bojarski et al. [3] at 1.85×10⁻⁴.
• The model's steering angle prediction was near-perfect (loss ≈ 10⁻⁵) across all configurations, indicating robustness in this control signal.
• The method demonstrated effective adaptation even when initialized far from demonstration distribution, highlighting resilience to domain shift.