[Paper Review] Differentiable Quality Diversity

Matthew C. Fontaine, Stefanos Nikolaidis|arXiv (Cornell University)|Jun 7, 2021

Machine Learning and Data Classification56 references30 citations

TL;DR

The paper defines Differentiable Quality Diversity (DQD) and presents MEGA, gradient-based MAP-Elites variants that leverage objective and measure gradients to achieve superior quality-diversity trade-offs in multiple domains, including StyleGAN latent space.

ABSTRACT

Quality diversity (QD) is a growing branch of stochastic optimization research that studies the problem of generating an archive of solutions that maximize a given objective function but are also diverse with respect to a set of specified measure functions. However, even when these functions are differentiable, QD algorithms treat them as "black boxes", ignoring gradient information. We present the differentiable quality diversity (DQD) problem, a special case of QD, where both the objective and measure functions are first order differentiable. We then present MAP-Elites via a Gradient Arborescence (MEGA), a DQD algorithm that leverages gradient information to efficiently explore the joint range of the objective and measure functions. Results in two QD benchmark domains and in searching the latent space of a StyleGAN show that MEGA significantly outperforms state-of-the-art QD algorithms, highlighting DQD's promise for efficient quality diversity optimization when gradient information is available. Source code is available at https://github.com/icaros-usc/dqd.

Motivation & Objective

• Formalize the differentiable quality diversity (DQD) problem where objective and measure functions are first-order differentiable.
• Develop gradient-based MAP-Elites variants (OMG-MEGA and CMA-MEGA) that leverage gradients to explore the joint objective-measure space.
• Show that gradient-based DQD significantly improves search efficiency across multiple benchmark domains.
• Demonstrate successful diverse, high-quality image generation in StyleGAN latent space using CMA-MEGA (Adam).

Proposed method

• Define DQD as a QD problem with differentiable objective f and differentiable measures m_i.
• Introduce gradient arborescence concepts to guide ascent on f while branching on m_i.
• Propose OMG-MEGA (Objective and Measure Gradient MAP-Elites via Gradient Arborescence) which perturbs solutions using a gradient-linear combination of ∇f and ∇m_i.
• Propose CMA-MEGA (CMA-MAP-Elites via Gradient Arborescence) which learns a distribution over coefficient vectors c to maximize the QD objective, adapting CMA-ES parameters.
• Provide baselines OG-MAP-Elites and OG-MAP-Elites (line) to contrast gradient-based and gradient-free exploration.
• Apply to three domains (Linear Projection, Arm Repertoire, Latent Space Illumination) and a StyleGAN-LSI setup with CLIP-based differentiable objectives/measures.
• Present Algorithm 1 detailing CMA-MEGA and discuss a natural-gradient interpretation of CMA-MEGA.

Experimental results

Research questions

• RQ1 Can gradients of the objective and measures be effectively integrated into a MAP-Elites style archive to improve quality-diversity optimization?
• RQ2 How do gradient-based DQD methods compare to derivative-free QD baselines across benchmark domains and in generative latent spaces?
• RQ3 What is the impact of adaptively selecting gradient step coefficients (e.g., CMA-MEGA with Adam) on search performance, especially in ill-conditioned or high-variance domains?
• RQ4 To what extent can differentiable QD optimize latent-space generative models (StyleGAN) to produce diverse, high-quality images meeting differentiable prompts?

Key findings

• Gradient-based DQD (OMG-MEGA and CMA-MEGA) substantially outperforms derivative-free baselines in QD-score and coverage across sphere, Rastrigin, and Arm Repertoire domains.
• CMA-MEGA (Adam) achieves the best performance overall, especially in the Latent Space Illumination (LSI) domain where conditioning benefits from adaptive gradient coefficients.
• Gradient information accelerates exploration of the measure space while ascending the objective, leading to richer archives and higher-quality diverse solutions.
• In LSI with StyleGAN-CLIP, CMA-MEGA (Adam) significantly outperforms other methods, though CMA-MEGA without Adam underperforms due to latent space conditioning.
• OG-MAP-Elites variants perform poorly in some domains, highlighting the advantage of combining gradient-guided steps with archiving mechanics.
• Table 1 shows CMA-MEGA and CMA-MEGA (Adam) achieving high QD-scores and coverage, particularly in the LSI domain (e.g., CMA-MEGA (Adam): QD-score 73.82–74.18%, coverage ~30–31% in LSI).