[Paper Review] Theoretical foundation for CMA-ES from information geometric perspective

Youhei Akimoto, Yuichi Nagata+2 more|arXiv (Cornell University)|Jun 4, 2012

Face and Expression Recognition21 references4 citations

TL;DR

This paper establishes a theoretical foundation for the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) using information geometry, showing that its parameter updates are equivalent to natural gradient learning for maximizing expected fitness. The key result is that CMA-ES implicitly performs natural gradient ascent without explicitly computing the Fisher information matrix inverse, justifying its default learning rates and linking it to generalized expectation-maximization frameworks.

ABSTRACT

This paper explores the theoretical basis of the covariance matrix adaptation evolution strategy (CMA-ES) from the information geometry viewpoint. To establish a theoretical foundation for the CMA-ES, we focus on a geometric structure of a Riemannian manifold of probability distributions equipped with the Fisher metric. We define a function on the manifold which is the expectation of fitness over the sampling distribution, and regard the goal of update of the parameters of sampling distribution in the CMA-ES as maximization of the expected fitness. We investigate the steepest ascent learning for the expected fitness maximization, where the steepest ascent direction is given by the natural gradient, which is the product of the inverse of the Fisher information matrix and the conventional gradient of the function. Our first result is that we can obtain under some types of parameterization of multivariate normal distribution the natural gradient of the expected fitness without the need for inversion of the Fisher information matrix. We find that the update of the distribution parameters in the CMA-ES is the same as natural gradient learning for expected fitness maximization. Our second result is that we derive the range of learning rates such that a step in the direction of the exact natural gradient improves the parameters in the expected fitness. We see from the close relation between the CMA-ES and natural gradient learning that the default setting of learning rates in the CMA-ES seems suitable in terms of monotone improvement in expected fitness. Then, we discuss the relation to the expectation-maximization framework and provide an information geometric interpretation of the CMA-ES.

Motivation & Objective

• To provide a theoretical justification for the CMA-ES algorithm using information geometry.
• To investigate the relationship between CMA-ES and natural gradient learning on a Riemannian manifold of probability distributions.
• To explain why the default learning rates in CMA-ES are effective in ensuring monotonic improvement in expected fitness.
• To interpret CMA-ES as a generalized expectation-maximization (GEM) algorithm from an information geometric perspective.
• To establish a connection between CMA-ES and the natural gradient method, particularly in terms of parameter update dynamics.

Proposed method

• The authors model the sampling distribution as a multivariate normal distribution on a Riemannian manifold equipped with the Fisher metric.
• They define the expected fitness as a function on this manifold and analyze steepest ascent via the natural gradient, which is the product of the inverse Fisher information matrix and the conventional gradient.
• Under a specific parameterization of the multivariate normal distribution, the natural gradient of the expected fitness is computed without explicitly inverting the Fisher information matrix.
• The CMA-ES parameter update rule is shown to be mathematically equivalent to natural gradient learning with sample-based estimation of the natural gradient.
• The authors derive a range of learning rates that guarantee monotonic improvement in expected fitness when following the exact natural gradient direction.
• They compare CMA-ES to the expectation-maximization (EM) framework and interpret it as a generalized EM (GEM) algorithm that considers both KL divergence and fitness improvement terms.

Experimental results

Research questions

• RQ1 How does the CMA-ES parameter update rule relate to natural gradient learning on the manifold of multivariate normal distributions?
• RQ2 Can the natural gradient of the expected fitness be computed without explicitly inverting the Fisher information matrix under certain parameterizations?
• RQ3 What range of learning rates ensures monotonic improvement in expected fitness when following the exact natural gradient?
• RQ4 How does the CMA-ES compare to the EM and GEM frameworks in terms of information geometric structure and convergence properties?
• RQ5 What is the information geometric interpretation of the CMA-ES in relation to the target distribution defined by fitness-weighted sampling?

Key findings

• The CMA-ES parameter update rule is mathematically equivalent to natural gradient learning for maximizing the expected fitness under the sampling distribution.
• Under a specific parameterization of the multivariate normal distribution, the natural gradient of the expected fitness can be estimated directly from samples without computing the inverse of the Fisher information matrix.
• The default learning rates used in CMA-ES are justified as being within a range that ensures monotonic improvement in expected fitness when following the exact natural gradient.
• The CMA-ES can be interpreted as a generalized EM (GEM) algorithm that improves both the KL divergence to the target distribution and the expected fitness, unlike standard EM which only maximizes one component.
• The information geometric view reveals that CMA-ES balances two terms in fitness improvement: the change in the sampling distribution and the change in the fitness-weighted target distribution, offering a more robust update than standard EM.
• The natural gradient formulation provides a principled foundation for CMA-ES and suggests potential for improved convergence theory and more efficient parameter updates in future work.