[Paper Review] Biomedical image analysis competitions: The state of current participation practice

Matthias Eisenmann, Annika Reinke+354 more|arXiv (Cornell University)|Dec 16, 2022

Radiomics and Machine Learning in Medical Imaging17 citations

TL;DR

This study investigates current practices in biomedical image analysis competitions through a survey of 80 IEEE ISBI and MICCAI 2021 challenge participants. It reveals that knowledge exchange drives participation (70%), deep learning dominates (94% of solutions), and key challenges include time constraints (32% report insufficient time) and data size (43% report processing limitations), with patch-based training and downsampling being common solutions.

ABSTRACT

The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

Motivation & Objective

• To understand the current state of participation practices in biomedical image analysis competitions.
• To identify the primary motivations for participating in benchmarking challenges.
• To analyze methodological choices, algorithmic strategies, and technical bottlenecks in challenge development.
• To assess the role of validation practices, data handling, and infrastructure in competition performance.
• To evaluate the prevalence of deep learning, model ensembling, and postprocessing in submitted solutions.

Proposed method

• Conducted an international survey targeting all participants of 80 competitions hosted during IEEE ISBI 2021 and MICCAI 2021.
• Survey structured into five parts: team and task information, expertise and work environment, competition strategy, algorithm characteristics, and miscellaneous details.
• Collected data on motivation, time investment, infrastructure access, model architecture, data processing, and validation techniques.
• Analyzed responses from a median of 72% of challenge participants across 80 distinct competitions with dedicated leaderboards.
• Used descriptive statistics and cross-sectional analysis to summarize trends in model development, data handling, and validation practices.
• Applied the BIAS guideline for biomedical challenges to ensure methodological rigor in survey design and interpretation.

Experimental results

Research questions

• RQ1 What are the primary motivations for participation in biomedical image analysis competitions?
• RQ2 How much time do participants typically spend on method development, and what proportion report insufficient time?
• RQ3 What are the most common deep learning architectures and data processing strategies used in competition submissions?
• RQ4 To what extent are standard validation practices such as k-fold cross-validation and model ensembling employed?
• RQ5 What are the main technical bottlenecks reported by participants, particularly regarding data size and infrastructure?

Key findings

• Knowledge exchange was the primary motivation for 70% of participants, while prize money was a minor incentive (16%).
• A median of 80 working hours were spent on method development, yet 32% of participants reported insufficient time for development.
• 94% of all submitted solutions were based on deep learning, with 84% using standard architectures.
• 43% of respondents reported that image data were too large to process in full, with patch-based training (69%) and downsampling (37%) being the most common solutions.
• Only 37% of participants used k-fold cross-validation on the training set, and 50% performed model ensembling, with 61% using identical models and 39% using heterogeneous models.
• 48% of participants applied postprocessing steps to their predictions, indicating a common practice to refine results after inference.