[Paper Review] Simple random search provides a competitive approach to reinforcement learning

Horia Mania, Aurelia Guy+1 more|arXiv (Cornell University)|Mar 19, 2018

Reinforcement Learning in Robotics35 references169 citations

TL;DR

The authors show that a simple augmented random search in the parameter space of linear policies can match or exceed state-of-the-art sample efficiency on MuJoCo locomotion tasks, and is substantially more computation-efficient than Evolution Strategies. They also highlight high variance across seeds and hyperparameters in RL benchmarks.

ABSTRACT

A common belief in model-free reinforcement learning is that methods based on random search in the parameter space of policies exhibit significantly worse sample complexity than those that explore the space of actions. We dispel such beliefs by introducing a random search method for training static, linear policies for continuous control problems, matching state-of-the-art sample efficiency on the benchmark MuJoCo locomotion tasks. Our method also finds a nearly optimal controller for a challenging instance of the Linear Quadratic Regulator, a classical problem in control theory, when the dynamics are not known. Computationally, our random search algorithm is at least 15 times more efficient than the fastest competing model-free methods on these benchmarks. We take advantage of this computational efficiency to evaluate the performance of our method over hundreds of random seeds and many different hyperparameter configurations for each benchmark task. Our simulations highlight a high variability in performance in these benchmark tasks, suggesting that commonly used estimations of sample efficiency do not adequately evaluate the performance of RL algorithms.

Motivation & Objective

• Demystify whether exploration in parameter space can be as effective as action-space exploration for model-free RL.
• Develop a minimal, derivative-free optimization method for training linear policies that is computationally efficient.
• Evaluate ARS on standard MuJoCo locomotion benchmarks and a difficult LQR instance to assess performance and robustness across seeds.
• Highlight the variability in RL performance across seeds and hyperparameters to inform benchmarking practices.

Proposed method

• Present a basic random search (BRS) baseline for derivative-free optimization in RL.
• Augment BRS with scaling by reward standard deviation, online state normalization, and discarding poorly performing directions (ARS).
• Introduce four ARS variants: V1, V1-t, V2, V2-t, with V2 including state whitening and V1/V2-t using top-direction selection.
• Use a parallel implementation with a shared noise table and independent rollouts to estimate gradients along random directions.
• Formulate an oracle model for RL to discuss sample complexity as the number of rollouts (queries) to the environment.
• Compare ARS against NG, TRPO, ES, PPO, A2C, CEM, and SAC on MuJoCo tasks, and analyze sample efficiency and wall-clock time.

Experimental results

Research questions

• RQ1 Can a simple random search in policy parameter space achieve competitive sample efficiency on continuous control tasks?
• RQ2 Do augmentations such as reward scaling, state normalization, and selection of top directions improve ARS performance?
• RQ3 How does ARS compare to mainstream RL methods in terms of sample efficiency and computational cost on MuJoCo benchmarks?
• RQ4 What is the impact of evaluation seed variability and hyperparameter sensitivity on RL benchmarking practices?
• RQ5 Do linear policies trained by ARS perform well on challenging control tasks and unknown-dynamics problems like a difficult LQR instance?

Key findings

• ARS matches or exceeds state-of-the-art sample efficiency on MuJoCo locomotion tasks with linear policies (no neural networks).
• ARS is at least 15x more computationally efficient than ES on Humanoid-v1 when reaching similar performance thresholds.
• ARS demonstrates high variance across seeds and hyperparameters, underscoring the need for extensive benchmarking with many trials.
• ARS V2 (with state normalization/whitening) solves Humanoid-v1 and improves performance on most MuJoCo tasks relative to V1.
• ARS can solve a difficult instance of the unknown-dynamics LQR problem to nearly optimal performance.
• Compared to a large set of baselines, ARS generally achieves favorable sample efficiency and competitive maximum rewards after 1e6 timesteps.