[Paper Review] Accelerated Quality-Diversity through Massive Parallelism

Bryan Lim, Maxime Allard+2 more|arXiv (Cornell University)|Feb 2, 2022

Metaheuristic Optimization Algorithms Research4 citations

TL;DR

This paper introduces QDax, an accelerated implementation of MAP-Elites that leverages massive parallelism on GPUs/TPUs to reduce QD algorithm runtimes from days to minutes with no performance loss. By using large batch sizes and hardware-optimized simulators, QDax achieves interactive-scale performance while maintaining solution quality across diverse benchmarks.

ABSTRACT

Quality-Diversity (QD) optimization algorithms are a well-known approach to generate large collections of diverse and high-quality solutions. However, derived from evolutionary computation, QD algorithms are population-based methods which are known to be data-inefficient and requires large amounts of computational resources. This makes QD algorithms slow when used in applications where solution evaluations are computationally costly. A common approach to speed up QD algorithms is to evaluate solutions in parallel, for instance by using physical simulators in robotics. Yet, this approach is limited to several dozen of parallel evaluations as most physics simulators can only be parallelized more with a greater number of CPUs. With recent advances in simulators that run on accelerators, thousands of evaluations can now be performed in parallel on single GPU/TPU. In this paper, we present QDax, an accelerated implementation of MAP-Elites which leverages massive parallelism on accelerators to make QD algorithms more accessible. We show that QD algorithms are ideal candidates to take advantage of progress in hardware acceleration. We demonstrate that QD algorithms can scale with massive parallelism to be run at interactive timescales without any significant effect on the performance. Results across standard optimization functions and four neuroevolution benchmark environments shows that experiment runtimes are reduced by two factors of magnitudes, turning days of computation into minutes. More surprising, we observe that reducing the number of generations by two orders of magnitude, and thus having significantly shorter lineage does not impact the performance of QD algorithms. These results show that QD can now benefit from hardware acceleration, which contributed significantly to the bloom of deep learning.

Motivation & Objective

• Address the computational bottleneck of Quality-Diversity (QD) algorithms, which require millions of evaluations and are slow when solution evaluations are costly.
• Overcome the limitations of CPU-based simulators that restrict parallelization to dozens of evaluations and suffer from communication overhead.
• Enable QD algorithms to scale with modern hardware accelerators (GPUs/TPUs) to achieve interactive timescales.
• Demonstrate that massive parallelism does not degrade QD performance, even with drastically reduced generations.
• Increase accessibility of QD algorithms to broader research communities, including those in emerging economies, by enabling fast execution on single cloud GPUs/TPUs.

Proposed method

• Implement QDax as a GPU/TPU-optimized variant of MAP-Elites, using massive batch evaluations to parallelize solution evaluations across thousands of instances.
• Integrate with modern physics simulators like Brax and IsaacGym that natively support GPU execution and enable thousands of parallel simulations.
• Leverage high-performance programming frameworks (e.g., JAX, PyTorch) to efficiently manage large-scale tensor operations and device memory.
• Use a fixed archive structure to store diverse, high-quality solutions across a behavior space, with behavior descriptors guiding the search.
• Apply large batch sizes (thousands of evaluations per generation) to drastically reduce the number of generations required without sacrificing solution quality.
• Optimize memory usage to handle large archives and complex policies within device memory constraints, though this remains a scaling limitation.

Experimental results

Research questions

• RQ1 Can QD algorithms benefit from massive parallelism on accelerators (GPUs/TPUs) to achieve interactive-scale runtime without performance degradation?
• RQ2 How does reducing the number of generations by two orders of magnitude affect the final performance of QD algorithms when using large batch sizes?
• RQ3 To what extent can modern GPU-accelerated simulators replace traditional CPU-based simulators in accelerating QD workflows?
• RQ4 Does the use of massive parallelism compromise the diversity or quality of solutions found by QD algorithms?
• RQ5 Can QDax make QD algorithms more accessible to researchers without access to large CPU clusters or high-performance computing infrastructure?

Key findings

• QDax reduces runtime of QD algorithms by two orders of magnitude, cutting computation time from days to minutes on a single GPU/TPU.
• Despite reducing the number of generations by 100×, the final performance of QDax remains comparable to baseline QD algorithms, indicating robustness to reduced lineage length.
• The use of large batch sizes on accelerators enables thousands of parallel evaluations, making QD algorithms viable for interactive exploration and real-time feedback.
• QDax maintains solution diversity and quality across standard optimization functions and four neuroevolution benchmark environments, including complex RL tasks.
• The performance of QDax is not degraded by hardware acceleration, demonstrating that massive parallelism can be leveraged without trade-offs in solution quality.
• The environmental impact of QDax is reduced due to shorter training times—estimated emissions were calculated using the Machine Learning Impact calculator, supporting carbon accountability.