[Paper Review] Over-optimism in benchmark studies and the multiplicity of design and analysis options when interpreting their results

Christina Nießl, Moritz Herrmann+3 more|arXiv (Cornell University)|Jun 4, 2021

Meta-analysis and systematic reviewsDecision Sciences58 references38 citations

TL;DR

This paper demonstrates how design and analysis choices in benchmark studies—such as data set selection, performance measures, and aggregation methods—lead to highly variable method rankings, fostering over-optimism and biased conclusions. Using multidimensional unfolding, the authors propose a systematic framework to visualize and assess the impact of each choice, enhancing transparency and reliability in computational benchmarking.

ABSTRACT

In recent years, the need for neutral benchmark studies that focus on the comparison of methods from computational sciences has been increasingly recognised by the scientific community. While general advice on the design and analysis of neutral benchmark studies can be found in recent literature, certain amounts of flexibility always exist. This includes the choice of data sets and performance measures, the handling of missing performance values and the way the performance values are aggregated over the data sets. As a consequence of this flexibility, researchers may be concerned about how their choices affect the results or, in the worst case, may be tempted to engage in questionable research practices (e.g. the selective reporting of results or the post-hoc modification of design or analysis components) to fit their expectations or hopes. To raise awareness for this issue, we use an example benchmark study to illustrate how variable benchmark results can be when all possible combinations of a range of design and analysis options are considered. We then demonstrate how the impact of each choice on the results can be assessed using multidimensional unfolding. In conclusion, based on previous literature and on our illustrative example, we claim that the multiplicity of design and analysis options combined with questionable research practices lead to biased interpretations of benchmark results and to over-optimistic conclusions. This issue should be considered by computational researchers when designing and analysing their benchmark studies and by the scientific community in general in an effort towards more reliable benchmark results.

Motivation & Objective

• To highlight the risk of over-optimism in benchmark studies due to flexibility in design and analysis choices.
• To illustrate how varying combinations of design and analysis options can drastically alter method rankings.
• To propose a systematic framework using multidimensional unfolding to assess the impact of each choice on benchmark results.
• To advocate for greater transparency, sensitivity reporting, and pre-registration in benchmark studies to reduce bias.
• To promote reproducibility and reliability in computational research by encouraging code and data sharing.

Proposed method

• The authors use a real benchmark study as a case study to explore all possible combinations of design and analysis options.
• They apply multidimensional unfolding (MDS) to visualize the variability of method rankings across different design and analysis choices.
• The framework quantifies the impact of each individual choice (e.g., performance measure, data set subset) on the resulting rankings.
• The method enables researchers to identify which choices most strongly influence the outcome, allowing for targeted justification.
• The approach supports sensitivity analysis by graphically representing how rankings shift under alternative configurations.
• The framework is designed to be integrated into benchmark studies to improve transparency and reduce selective reporting.

Experimental results

Research questions

• RQ1 How do different combinations of design and analysis choices affect method rankings in benchmark studies?
• RQ2 To what extent can the same benchmark data produce divergent conclusions based on methodological choices?
• RQ3 Which design and analysis choices have the most substantial impact on the final ranking of methods?
• RQ4 How can researchers systematically assess the influence of each choice on benchmark outcomes?
• RQ5 What strategies can reduce over-optimism and improve the reliability of benchmark results?

Key findings

• The same benchmark data can yield vastly different method rankings depending on the choice of performance measures, data set subsets, and aggregation methods.
• The use of multidimensional unfolding effectively visualizes the sensitivity of rankings to design and analysis decisions, revealing critical choices that drive results.
• Certain choices—such as the selection of performance measures or data set groups—have a disproportionately large impact on method rankings.
• The framework enables researchers to identify and justify key decisions, reducing the risk of post-hoc rationalization and selective reporting.
• The study demonstrates that over-optimism in benchmark results is a real risk when researchers unconsciously favor choices that support their expectations.
• The authors conclude that transparency, sensitivity analysis, and code/data sharing are essential to improve the reliability and reproducibility of benchmark studies.