[论文解读] The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards

Sarah Holland, Ahmed Hosny+3 more|arXiv (Cornell University)|May 9, 2018

Nutritional Studies and Diet参考文献 21被引用 111

一句话总结

数据集营养标签提供一个灵活、标准化的定性与定量模块框架，用于在AI模型开发之前评估数据质量，并在ProPublica Dollars for Docs数据集的开源原型中得到证明。

ABSTRACT

Artificial intelligence (AI) systems built on incomplete or biased data will often exhibit problematic outcomes. Current methods of data analysis, particularly before model development, are costly and not standardized. The Dataset Nutrition Label (the Label) is a diagnostic framework that lowers the barrier to standardized data analysis by providing a distilled yet comprehensive overview of dataset "ingredients" before AI model development. Building a Label that can be applied across domains and data types requires that the framework itself be flexible and adaptable; as such, the Label is comprised of diverse qualitative and quantitative modules generated through multiple statistical and probabilistic modelling backends, but displayed in a standardized format. To demonstrate and advance this concept, we generated and published an open source prototype with seven sample modules on the ProPublica Dollars for Docs dataset. The benefits of the Label are manyfold. For data specialists, the Label will drive more robust data analysis practices, provide an efficient way to select the best dataset for their purposes, and increase the overall quality of AI models as a result of more robust training datasets and the ability to check for issues at the time of model development. For those building and publishing datasets, the Label creates an expectation of explanation, which will drive better data collection practices. We also explore the limitations of the Label, including the challenges of generalizing across diverse datasets, and the risk of using "ground truth" data as a comparison dataset. We discuss ways to move forward given the limitations identified. Lastly, we lay out future directions for the Dataset Nutrition Label project, including research and public policy agendas to further advance consideration of the concept.

研究动机与目标

• 推动对标准化数据分析的需求，以防止因数据不完整而导致的AI结果偏差。
• 定义一个可跨领域及数据类型应用的灵活框架。
• 在模型开发之前提供一个提炼的、标准化的数据集成成分概览。
• 通过一个开源原型演示该概念并讨论对数据收集和分析的影响。

提出的方法

• 提出数据集营养标签作为一个诊断框架，结合定性和定量模块。
• 结合多种统计与概率建模后端来生成标签模块。
• 以标准化、领域无关的格式显示结果，便于解读。
• 发布一个包含七个示例模块实现的开源原型。
• 讨论局限性、跨多样数据集的可推广性以及未来方向。

实验结果

研究问题

• RQ1 如何设计一个灵活、领域无关的框架来总结数据集质量及其对AI任务的适用性？
• RQ2 哪种定性与定量模块及后端的组合最能传达数据质量问题？
• RQ3 在指导数据集选择和模型开发时使用标签的影响与局限性是什么？

主要发现

• 该标签旨在推动数据专业人员进行更稳健的数据分析实践。
• 该框架对数据集发布方明确的解释期望，从而可能改进数据收集实践。
• 该标签使数据集选择能够更高效地与特定建模需求和质量考量保持一致。
• 开源原型展示了该概念并支持采纳与社区贡献。
• 局限性包括在多样数据集上的推广挑战以及将真实数据作为参考基准的相关风险。