[Paper Review] Benchmarking in Optimization: Best Practice and Open Issues

Thomas Bartz–Beielstein, Carola Doerr+15 more|arXiv (Cornell University)|Jul 7, 2020

Advanced Multi-Objective Optimization Algorithms213 references80 citations

TL;DR

A comprehensive survey outlining best-practice guidelines for benchmarking optimization algorithms, covering goals, problem sets, algorithms, performance measures, analysis, design, presentation, and reproducibility, with open issues and a path toward periodic updates.

ABSTRACT

This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.

Motivation & Objective

• Clarify the goals and motivation behind benchmarking studies in optimization.
• Identify desirable characteristics and evaluation criteria for problem sets and algorithms.
• Provide guidelines for performance measurement, data analysis, experimental design, and result presentation.
• Address reproducibility and data sharing challenges in benchmarking.
• Offer a framework that can evolve with the field through periodic updates.

Proposed method

• Synthesize recommendations from a diverse set of researchers across global institutions.
• Organize the discussion around eight core benchmarking topics: goals, problems, algorithms, performance, analysis, design, presentation, and reproducibility.
• Map practical best-practice examples to each topic and highlight open issues for future work.

Experimental results

Research questions

• RQ1 What are the main goals driving benchmarking studies in optimization?
• RQ2 How should problem instances be selected, evaluated, and kept representative and diverse?
• RQ3 What constitutes robust and informative performance measures and analyses for benchmarking?
• RQ4 How should experiments be designed and reported to guarantee reproducibility and comparability?
• RQ5 What open issues remain in benchmarking best practices and how should they be addressed in future updates?

Key findings

• Benchmarking in optimization should clearly state its goals, as these shape problem choices, algorithms, performance criteria, and statistics.
• Problem sets must be diverse, representative, scalable, and sometimes have known ground truths or best-known performances.
• Algorithm tuning and understanding require careful experimental design and robust statistical tools, with attention to parameter robustness.
• Performance extrapolation and training-like uses (e.g., AutoML) are viable but require careful handling of problem features and instance selection.
• Reproducibility hinges on precise algorithm/problem descriptions, seeds, data formats, and shared data repositories; standardization is an open issue.
• Benchmarking serves as a bridge between theory and practice and can inspire theoretical insights and algorithm development.