[Paper Review] Datasheets for AI and medical datasets (DAIMS): a data validation and documentation framework before machine learning analysis in medical research

Ramtin Zargari Marandi, Anne Svane Frahm+1 more|ArXiv.org|Jan 23, 2025

Artificial Intelligence in Healthcare3 citations

TL;DR

DAIMS extends the Datasheets for Datasets framework to AI and medical data, offering a 24-item standardization checklist, a supporting software tool, an extended documentation form, a data dictionary table, and a workflow flowchart to guide ML analyses in medical research.

ABSTRACT

Despite progresses in data engineering, there are areas with limited consistencies across data validation and documentation procedures causing confusions and technical problems in research involving machine learning. There have been progresses by introducing frameworks like "Datasheets for Datasets", however there are areas for improvements to prepare datasets, ready for ML pipelines. Here, we extend the framework to "Datasheets for AI and medical datasets - DAIMS." Our publicly available solution, DAIMS, provides a checklist including data standardization requirements, a software tool to assist the process of the data preparation, an extended form for data documentation and pose research questions, a table as data dictionary, and a flowchart to suggest ML analyses to address the research questions. The checklist consists of 24 common data standardization requirements, where the tool checks and validate a subset of them. In addition, we provided a flowchart mapping research questions to suggested ML methods. DAIMS can serve as a reference for standardizing datasets and a roadmap for researchers aiming to apply effective ML techniques in their medical research endeavors. DAIMS is available on GitHub and as an online app to automate key aspects of dataset evaluation, facilitating efficient preparation of datasets for ML studies.

Motivation & Objective

• Motivate the need for standardized data validation and documentation in medical ML research.
• Extend existing Datasheets for Datasets principles to AI and medical datasets (DAIMS).
• Provide a practical, publicly available toolkit to standardize, document, and guide ML analyses on medical data.

Proposed method

• Offer a 24-item data standardization checklist with automated subset validation.
• Develop a software tool to assist data preparation and validation.
• Provide an extended data documentation form and a data dictionary table.
• Include a flowchart mapping research questions to suggested ML methods.

Experimental results

Research questions

• RQ1 What data standardization requirements are most critical for preparing medical datasets for ML analyses?
• RQ2 How can a validation tool and documentation framework improve reproducibility and quality of ML workflows in medical research?
• RQ3 Can a flowchart effectively guide researchers from research questions to appropriate ML methods in medical contexts?

Key findings

• DAIMS delivers a publicly available checklist covering 24 common data standardization requirements.
• The solution includes a software tool that automates key aspects of dataset preparation and validation.
• An extended documentation form and data dictionary table are provided to improve data understandability and traceability.
• A flowchart maps research questions to suggested ML analyses to address them.
• DAIMS can serve as a reference for standardizing datasets and as a roadmap for applying ML techniques in medical research.