[Paper Review] A Critical Field Guide for Working with Machine Learning Datasets

Sarah Ciston, Mike Ananny+1 more|ArXiv.org|Jan 26, 2025

Big Data Technologies and Applications5 citations

TL;DR

The paper offers practical guidance for conscientious dataset stewardship across the lifecycle of ML datasets, blending critical AI theory with applied data science concepts to help researchers, journalists, artists, and developers work more responsibly with data.

ABSTRACT

Machine learning datasets are powerful but unwieldy. Despite the fact that large datasets commonly contain problematic material--whether from a technical, legal, or ethical perspective--datasets are valuable resources when handled carefully and critically. A Critical Field Guide for Working with Machine Learning Datasets suggests practical guidance for conscientious dataset stewardship. It offers questions, suggestions, strategies, and resources for working with existing machine learning datasets at every phase of their lifecycle. It combines critical AI theories and applied data science concepts, explained in accessible language. Equipped with this understanding, students, journalists, artists, researchers, and developers can be more capable of avoiding the problems unique to datasets. They can also construct more reliable, robust solutions, or even explore new ways of thinking with machine learning datasets that are more critical and conscientious.

Motivation & Objective

• Motivate the need for conscientious stewardship of machine learning datasets due to technical, legal, and ethical concerns in large datasets.
• Present practical guidance, questions, strategies, and resources for working with datasets at every lifecycle stage.
• Bridge critical AI theories with applied data science concepts in accessible language to empower diverse stakeholders.

Proposed method

• Provide a structured set of questions, suggestions, and strategies for dataset work across lifecycle stages.
• Offer resources and practical guidance aimed at avoiding common problems unique to datasets.
• Synthesize critical AI theory with applied data science concepts in accessible language for broad audiences.

Experimental results

Research questions

• RQ1 What practical questions and strategies can guide conscientious dataset stewardship across the ML data lifecycle?
• RQ2 What resources and approaches help various stakeholders avoid common data-related problems in ML pipelines?
• RQ3 How can critical AI theory be translated into accessible guidance for non-technical audiences working with datasets?
• RQ4 How can the guidance improve reliability, robustness, and ethical considerations in ML datasets?

Key findings

• Provides practical guidance, questions, strategies, and resources for conscientious dataset stewardship.
• Synthesizes critical AI theories with applied data science concepts in accessible language.
• Aims to enable students, journalists, artists, researchers, and developers to avoid problems unique to datasets and to build more reliable, robust solutions.