[Paper Review] EPOpt: Learning Robust Neural Network Policies Using Model Ensembles

Aravind Rajeswaran, Sarvjeet Ghotra+2 more|arXiv (Cornell University)|Oct 5, 2016

Reinforcement Learning in Robotics30 references144 citations

TL;DR

EPOpt trains robust neural policies by adversarially training over an ensemble of simulated models and adapting the source distribution with target-domain data.

ABSTRACT

Sample complexity and safety are major challenges when learning policies with reinforcement learning for real-world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model-based methods where the real-world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.

Motivation & Objective

• Motivate robust reinforcement learning for physical control under model misspecification and safety concerns.
• Propose a method to train policies that generalize across a distribution of source models using ensemble training.
• Introduce adaptation of the source model distribution using target-domain data to better approximate the target dynamics.

Proposed method

• Use an ensemble of source-domain models sampled from a parameter distribution to generate trajectories for policy updates.
• Optimize for a CVaR (epsilon-percentile) objective to focus learning on the worst-performing models in the ensemble.
• Employ a batch policy optimization subroutine (TRPO-based) to update the policy using the worst epsilon-fraction of trajectories.
• Adapt the source-domain distribution via approximate Bayesian updates using target-domain trajectories to refine model parameters.
• Optionally, apply importance sampling to reweight model samples during adaptation when the target domain differs substantially.

Experimental results

Research questions

• RQ1 Does training on a distribution of models (ensemble) improve policy robustness to model mismatch compared to single-model training?
• RQ2 How does the epsilon-CVaR based EPOpt variant affect direct-transfer performance to the target domain?
• RQ3 Can EPOpt learn policies that are robust to unmodeled effects not captured by the source-domain ensemble?
• RQ4 How efficiently can the source distribution be adapted to a target domain with limited target-domain data?
• RQ5 What are the comparative benefits of model adaptation versus standard maximum-likelihood model selection in Bayesian RL for transfer learning?

Key findings

• Policies trained with EPOpt-ε generalize across a wide range of model instances better than single-model TRPO on the Hopper and Half-Ccheetah benchmarks.
• EPOpt(0.1) yields highly robust policies with strong direct-transfer performance across varied model parameters.
• EPOpt is robust to unmodeled effects when the source domain includes varied parameters, though robustness improves when mass is included in the source distribution.
• Model adaptation can align the source distribution to the target domain with relatively small target-domain data, improving target performance over time.
• Direct-transfer performance of EPOpt is not significantly degraded by adopting a more conservative, robust strategy.