[Paper Review] Multi-objective Analysis of MAP-Elites Performance

Eivind Samuelsen, Kyrre Glette|arXiv (Cornell University)|Mar 14, 2018

Reinforcement Learning in Robotics17 references3 citations

TL;DR

This paper proposes a multi-objective performance analysis method for MAP-Elites that combines ordinal effect sizes (Cliff’s delta) with Pareto dominance to quantitatively compare algorithm performance across multiple, differently scaled metrics. The method reveals that mutation magnitude has a stronger impact on outcomes than mutation type or map resolution, with σ = 0.4 yielding the best trade-off between precision and coverage across four robot morphologies.

ABSTRACT

In certain complex optimization tasks, it becomes necessary to use multiple measures to characterize the performance of different algorithms. This paper presents a method that combines ordinal effect sizes with Pareto dominance to analyze such cases. Since the method is ordinal, it can also generalize across different optimization tasks even when the performance measurements are differently scaled. Through a case study, we show that this method can discover and quantify relations that would be difficult to deduce using a conventional measure-by-measure analysis. This case study applies the method to the evolution of robot controller repertoires using the MAP-Elites algorithm. Here, we analyze the search performance across a large set of parametrizations; varying mutation size and operator type, as well as map resolution, across four different robot morphologies. We show that the average magnitude of mutations has a bigger effect on outcomes than their precise distributions.

Motivation & Objective

• To address the challenge of comparing MAP-Elites performance across multiple, differently scaled performance metrics such as precision, coverage, and reliability.
• To develop a robust, generalizable method for analyzing algorithmic trade-offs in Quality Diversity (QD) search that accounts for stochasticity and scaling differences.
• To evaluate the impact of key MAP-Elites hyperparameters—mutation size, mutation type, and map resolution—on performance across diverse robot morphologies.
• To demonstrate that ordinal effect size analysis provides clearer, more generalizable insights than conventional measure-by-measure or boxplot-based comparisons.

Proposed method

• The method uses Cliff’s delta, an ordinal non-parametric effect size, to quantify the degree to which one parameter configuration’s performance dominates another across multiple runs.
• It applies Pareto dominance to compare performance across multiple objectives (precision, coverage, reliability), treating each run as a stochastic outcome.
• Confidence intervals for Cliff’s delta are computed via bootstrapping to test for statistical significance (α = 0.01), with overlap with zero indicating no significant difference.
• Performance is visualized using parametric plots that trace mean performance over evaluation count, with shaded regions indicating dominance by at least half the runs.
• The approach generalizes across robot morphologies by aggregating results across different scaling regimes, enabling cross-configuration comparisons.
• The method is applied to a case study on evolving robot controller repertoires using MAP-Elites, varying mutation size (σ), mutation type (all vs. partial parameters), and map resolution (5×5 to 9×9).

Experimental results

Research questions

• RQ1 How do different mutation sizes (σ) affect the precision and coverage of MAP-Elites in evolving robot controller repertoires?
• RQ2 What is the relative impact of mutation type (all parameters vs. probabilistic perturbation) on MAP-Elites performance across diverse robot morphologies?
• RQ3 How does map resolution influence the trade-off between coverage and precision in the behavior space?
• RQ4 Can ordinal effect size analysis with Pareto dominance provide more generalizable and interpretable insights than traditional measure-by-measure or boxplot-based analysis?

Key findings

• The mutation size σ = 0.4 yields the best overall performance, with the largest half-dominating area and superior trade-off between precision and coverage across all robot morphologies.
• Mutation size σ = 0.2 and σ = 0.8 show equivalent performance, both outperforming σ = 0.05 and σ = 0.1, which are significantly worse.
• Smaller map resolutions (5×5) consistently outperform larger ones (7×7 and 9×9), with a strong preference for 5×5 over 7×7 and 7×7 over 9×9.
• The effect size for mutation type is small and not statistically significant, suggesting no meaningful difference between mutating all parameters versus only some.
• The parametric plots show exponential convergence trends, with evenly spaced marks indicating consistent progress over evaluation count.
• The method successfully generalizes performance insights across four robot morphologies, revealing consistent trade-offs despite differing scaling of performance metrics.