[Paper Review] Environment Probing Interaction Policies

Wenxuan Zhou, Lerrel Pinto+1 more|arXiv (Cornell University)|Jul 26, 2019

Reinforcement Learning in Robotics33 citations

TL;DR

The paper introduces Environment-Probing Interaction (EPI) policies that first probe a new environment to extract an environment embedding, which is then used by a task-specific policy to improve generalization to unseen environments. EPI training uses a transition-prediction-based reward to optimize the probing policy.

ABSTRACT

A key challenge in reinforcement learning (RL) is environment generalization: a policy trained to solve a task in one environment often fails to solve the same task in a slightly different test environment. A common approach to improve inter-environment transfer is to learn policies that are invariant to the distribution of testing environments. However, we argue that instead of being invariant, the policy should identify the specific nuances of an environment and exploit them to achieve better performance. In this work, we propose the 'Environment-Probing' Interaction (EPI) policy, a policy that probes a new environment to extract an implicit understanding of that environment's behavior. Once this environment-specific information is obtained, it is used as an additional input to a task-specific policy that can now perform environment-conditioned actions to solve a task. To learn these EPI-policies, we present a reward function based on transition predictability. Specifically, a higher reward is given if the trajectory generated by the EPI-policy can be used to better predict transitions. We experimentally show that EPI-conditioned task-specific policies significantly outperform commonly used policy generalization methods on novel testing environments.

Motivation & Objective

• Motivate and address environment generalization in RL beyond invariant policies and explicit system identification.
• Propose a separate environment-probing policy (EPI-policy) to extract environment information.
• Learn environment embeddings from probing interactions and condition a task policy on this embedding.
• Show that EPI-conditioned policies outperform standard baselines on novel OpenAI Gym MuJoCo tasks.

Proposed method

• Define two policies: an Environment-Probing Interaction (EPI) policy and a task-specific policy.
• Train transition predictors: a vanilla predictor f(s,a)->s' and an EPI-conditioned predictor f_epi(s,a; psi(tau_epi)); psi(tau_epi) is an embedding of the EPI trajectory.
• Use a reward R_p(pi_epi) equal to E[ L_epi_pred - L_pred ], encouraging trajectories that improve environment transition prediction.
• Interleave training of the EPI policy and the predictors (interleaved/training data via Vine method and optional separation loss).
• After learning, run EPI to obtain an environment embedding, then condition the task policy pi_task on this embedding and train pi_task with TRPO.
• Evaluate on Hopper and Striker with randomized environment parameters (mass, damping, friction).
• Compare against baselines including Simple, Invariant, Oracle, Random/History/Recurrent/System-Id/Direct Reward policies.

Experimental results

Research questions

• RQ1 Can an environment-probing policy extract informative, low-dimensional environment embeddings that help a separate task policy generalize to unseen environments?
• RQ2 Is a prediction-based reward for probing trajectories a better driver than curiosity-based or random exploration for this purpose?
• RQ3 How does an EPI-conditioned policy compare to conventional generalization approaches (domain randomization, system identification, recurrent policies) in novel environments?

Key findings

• EPI-conditioned task policies outperform standard baselines on unseen testing environments (Hopper and Striker).
• In Hopper, EPI achieves 1303±173 reward, at least 14.0% better than baselines.
• In Striker, EPI achieves 0.162±0.015 final distance, at least 37.5% more accurate than baselines.
• The EPI-embedding correlates with environment parameters, indicating the method disentangles environment-induced transition differences.
• Ablation tests show Vine data and regularization improve performance; without them, EPI still beats most baselines but with reduced gains.
• Oracle policy (full environment parameters) remains the upper bound (1474±205 reward; 0.133±0.034 distance).