[Paper Review] The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards

Sarah Holland, Ahmed Hosny+3 more|arXiv (Cornell University)|May 9, 2018

Nutritional Studies and DietMedicine21 references111 citations

TL;DR

The Dataset Nutrition Label provides a flexible, standardized framework of qualitative and quantitative modules to assess data quality before AI model development, demonstrated with an open-source prototype on the ProPublica Dollars for Docs dataset.

ABSTRACT

Artificial intelligence (AI) systems built on incomplete or biased data will often exhibit problematic outcomes. Current methods of data analysis, particularly before model development, are costly and not standardized. The Dataset Nutrition Label (the Label) is a diagnostic framework that lowers the barrier to standardized data analysis by providing a distilled yet comprehensive overview of dataset "ingredients" before AI model development. Building a Label that can be applied across domains and data types requires that the framework itself be flexible and adaptable; as such, the Label is comprised of diverse qualitative and quantitative modules generated through multiple statistical and probabilistic modelling backends, but displayed in a standardized format. To demonstrate and advance this concept, we generated and published an open source prototype with seven sample modules on the ProPublica Dollars for Docs dataset. The benefits of the Label are manyfold. For data specialists, the Label will drive more robust data analysis practices, provide an efficient way to select the best dataset for their purposes, and increase the overall quality of AI models as a result of more robust training datasets and the ability to check for issues at the time of model development. For those building and publishing datasets, the Label creates an expectation of explanation, which will drive better data collection practices. We also explore the limitations of the Label, including the challenges of generalizing across diverse datasets, and the risk of using "ground truth" data as a comparison dataset. We discuss ways to move forward given the limitations identified. Lastly, we lay out future directions for the Dataset Nutrition Label project, including research and public policy agendas to further advance consideration of the concept.

Motivation & Objective

• Motivate the need for standardized data analysis to prevent AI outcomes biased by incomplete data.
• Define a flexible framework that can be applied across domains and data types.
• Provide a distilled, standardized overview of dataset ingredients before model development.
• Demonstrate the concept with an open-source prototype and discuss implications for data collection and analysis.

Proposed method

• Propose the Dataset Nutrition Label as a diagnostic framework combining qualitative and quantitative modules.
• Incorporate multiple statistical and probabilistic modelling backends to generate Label modules.
• Display the results in a standardized, domain-agnostic format for ease of interpretation.
• Publish an open-source prototype with seven sample modules implemented.
• Discuss limitations, generalizability across diverse datasets, and future directions.

Experimental results

Research questions

• RQ1 How can a flexible, domain-agnostic framework be designed to summarize dataset quality and suitability for AI tasks?
• RQ2 What combination of qualitative and quantitative modules and backends best conveys data quality issues?
• RQ3 What are the impacts and limitations of using the Label in guiding dataset selection and model development?

Key findings

• The Label is positioned to drive more robust data analysis practices for data specialists.
• The framework creates an explicit expectation of explanation for dataset publishers, potentially improving data collection practices.
• The Label enables more efficient dataset selection aligned with specific modeling needs and quality considerations.
• The open-source prototype demonstrates the concept and supports adoption and community contribution.
• Limitations include challenges in generalizing across diverse datasets and risks related to using ground truth data as a reference.