[Paper Review] Healthsheet: Development of a Transparency Artifact for Health Datasets

Negar Rostamzadeh, Diana Mincu+8 more|arXiv (Cornell University)|Feb 26, 2022

Mobile Health and mHealth Applications8 citations

TL;DR

This paper introduces Healthsheet, a healthcare-specific adaptation of the datasheets for datasets framework, to enhance transparency and accountability in ML-driven health research. Through interviews and case studies on EHR, clinical trial, and digital health datasets, the authors demonstrate that Healthsheet improves bias detection, supports ethical dataset evaluation, and promotes community-driven documentation for equitable ML in healthcare.

ABSTRACT

Machine learning (ML) approaches have demonstrated promising results in a wide range of healthcare applications. Data plays a crucial role in developing ML-based healthcare systems that directly affect people's lives. Many of the ethical issues surrounding the use of ML in healthcare stem from structural inequalities underlying the way we collect, use, and handle data. Developing guidelines to improve documentation practices regarding the creation, use, and maintenance of ML healthcare datasets is therefore of critical importance. In this work, we introduce Healthsheet, a contextualized adaptation of the original datasheet questionnaire ~\cite{gebru2018datasheets} for health-specific applications. Through a series of semi-structured interviews, we adapt the datasheets for healthcare data documentation. As part of the Healthsheet development process and to understand the obstacles researchers face in creating datasheets, we worked with three publicly-available healthcare datasets as our case studies, each with different types of structured data: Electronic health Records (EHR), clinical trial study data, and smartphone-based performance outcome measures. Our findings from the interviewee study and case studies show 1) that datasheets should be contextualized for healthcare, 2) that despite incentives to adopt accountability practices such as datasheets, there is a lack of consistency in the broader use of these practices 3) how the ML for health community views datasheets and particularly extit{Healthsheets} as diagnostic tool to surface the limitations and strength of datasets and 4) the relative importance of different fields in the datasheet to healthcare concerns.

Motivation & Objective

• Address the lack of standardized, ethical documentation for healthcare datasets used in machine learning.
• Identify barriers and inconsistencies in current data documentation practices within the ML for health community.
• Develop a contextually tailored datasheet framework that reflects the unique needs and challenges of health data.
• Enable dataset consumers to better assess data quality, bias, and representativeness through structured transparency.
• Foster community adoption and evolution of dataset documentation as a tool for ethical and accountable ML in healthcare.

Proposed method

• Adapted the original datasheets for datasets framework (Gebru et al., 2018) to create Healthsheet, a healthcare-specific transparency artifact.
• Conducted semi-structured interviews with researchers and data stewards to identify key documentation needs and challenges.
• Applied Healthsheet to three publicly available healthcare datasets: MIMIC-III (EHR), clinical trial data, and smartphone-based performance measures.
• Mapped core datasheet components—such as data collection, versioning, and creator demographics—to healthcare-specific concerns.
• Evaluated the feasibility and utility of Healthsheet through iterative case studies and stakeholder feedback.
• Proposed a community-driven model for maintaining and evolving Healthsheet across new dataset versions and use cases.

Experimental results

Research questions

• RQ1 How can dataset documentation be adapted to meet the unique ethical and structural challenges of healthcare data?
• RQ2 What are the primary obstacles researchers face in creating and maintaining comprehensive dataset documentation in healthcare?
• RQ3 How do dataset consumers perceive the value of transparency artifacts like Healthsheet in assessing data quality and bias?
• RQ4 Which components of a datasheet are most critical for ethical evaluation in health-related ML applications?
• RQ5 To what extent can Healthsheet serve as a diagnostic tool for identifying data limitations and strengths in real-world healthcare datasets?

Key findings

• Healthsheet effectively contextualizes general datasheet principles for healthcare, addressing domain-specific concerns such as data provenance, bias, and clinical relevance.
• Despite recognition of the value of transparency, there is significant inconsistency in current documentation practices across healthcare datasets.
• Researchers view Healthsheet as a diagnostic tool that enhances understanding of dataset limitations and strengths, particularly in identifying biases and data representativeness.
• The process of completing a Healthsheet revealed major gaps in available metadata, especially regarding data acquisition mechanisms and versioning schemes.
• Community verification and ongoing maintenance of Healthsheet entries are essential, as dataset owners are the only ones who can accurately report on data intent and provenance.
• The lack of standardized nomenclature for dataset versions and the absence of centralized documentation repositories hinder transparency and reproducibility in health ML research.