[Paper Review] Sampled Policy Gradient for Learning to Play the Game Agar.io

Anton Wiehe, Nil Stolt-Ansó+2 more|arXiv (Cornell University)|Sep 15, 2018

Reinforcement Learning in Robotics20 references6 citations

TL;DR

This paper introduces Sampled Policy Gradient (SPG), a novel offline actor-critic reinforcement learning algorithm that improves exploration in continuous control by sampling actions in the action space and using a critic to approximate the policy gradient. SPG matches DPG's performance without extensive sampling and outperforms Q-learning and CACLA in pellet collection, while all algorithms fail to defeat a greedy bot in combat scenarios.

ABSTRACT

In this paper, a new offline actor-critic learning algorithm is introduced: Sampled Policy Gradient (SPG). SPG samples in the action space to calculate an approximated policy gradient by using the critic to evaluate the samples. This sampling allows SPG to search the action-Q-value space more globally than deterministic policy gradient (DPG), enabling it to theoretically avoid more local optima. SPG is compared to Q-learning and the actor-critic algorithms CACLA and DPG in a pellet collection task and a self play environment in the game Agar.io. The online game Agar.io has become massively popular on the internet due to intuitive game design and the ability to instantly compete against players around the world. From the point of view of artificial intelligence this game is also very intriguing: The game has a continuous input and action space and allows to have diverse agents with complex strategies compete against each other. The experimental results show that Q-Learning and CACLA outperform a pre-programmed greedy bot in the pellet collection task, but all algorithms fail to outperform this bot in a fighting scenario. The SPG algorithm is analyzed to have great extendability through offline exploration and it matches DPG in performance even in its basic form without extensive sampling.

Motivation & Objective

• To develop a new off-policy actor-critic algorithm that enables more global exploration in continuous control environments.
• To compare SPG with established algorithms like Q-learning, CACLA, and DPG in a simplified Agar.io environment.
• To evaluate the impact of offline exploration and action sampling on policy gradient learning in complex, stochastic environments.
• To investigate why CACLA outperforms DPG and SPG in certain tasks despite using a state-value critic instead of a state-action value function.
• To explore the potential of SPG as a flexible, extensible framework for future RL research in continuous action spaces.

Proposed method

• SPG approximates the policy gradient by sampling multiple actions in the action space and using the critic to evaluate their expected returns.
• The algorithm uses a critic network to estimate the value of sampled actions, enabling a more global search over the action-Q-value space compared to deterministic policy gradients.
• It employs offline exploration via Gaussian sampling, storing the best sampled action per transition to improve training stability and performance.
• The actor is updated using the gradient of the critic’s output with respect to the sampled actions, combining elements of DPG and CACLA.
• Hyperparameters were tuned via coarse search, and all algorithms used the same Keras-based implementation and OpenAI’s experience replay buffer to ensure fair comparison.
• The method supports extensions such as distributional critics and hierarchical architectures, enhancing its adaptability to complex environments.

Experimental results

Research questions

• RQ1 Can sampled action-space exploration in SPG lead to better convergence and reduced local optima compared to deterministic policy gradients?
• RQ2 How does SPG compare in performance to Q-learning, CACLA, and DPG in both pellet collection and self-play scenarios in Agar.io?
• RQ3 Why does CACLA outperform DPG and SPG in the pellet collection task despite using a state-value critic instead of a state-action value function?
• RQ4 To what extent can offline exploration and best-action caching improve SPG’s learning efficiency and final performance?
• RQ5 Can SPG be extended effectively with advanced techniques like distributional critics or curiosity-driven exploration in future work?

Key findings

• SPG matches the final performance of DPG in the pellet collection task, despite learning more slowly initially, and achieves this without heavy sampling or architectural extensions.
• Q-learning outperforms actor-critic methods in pellet collection by leveraging a simpler, more stable learning mechanism, even though it discretizes the continuous action space.
• CACLA achieves the best performance among actor-critic algorithms due to its use of a state-value critic, suggesting that critic function choice significantly impacts learning stability and outcome.
• All algorithms fail to outperform a pre-programmed greedy bot in direct combat scenarios, indicating that current methods struggle with complex, high-stakes strategic interactions.
• The use of offline Gaussian sampling and storing the best sampled action per transition significantly improves SPG’s performance, making it a recommended default configuration.
• The analysis suggests that SPG can serve as a strong foundation for future extensions, particularly in environments requiring robust exploration and critic accuracy.