[Paper Review] A Total Error Framework for Digital Traces of Humans

Indira Sen, Fabian Floeck+3 more|arXiv (Cornell University)|Jul 18, 2019

Human Mobility and Location-Based Analysis108 references21 citations

TL;DR

This paper proposes a Total Error Framework for digital traces of humans, adapting the Total Survey Error Framework to diagnose, understand, and mitigate errors in social science research using digital data. By systematically categorizing errors across data collection, processing, and analysis stages, the framework enables researchers to improve the validity and reliability of inferences drawn from digital traces.

ABSTRACT

The interactions and activities of hundreds of millions of people worldwide are recorded as digital traces every single day. When pulled together, these data offer increasingly comprehensive pictures of both individuals and groups interacting on different platforms, but they also allow inferences about broader target populations beyond those platforms, representing an enormous potential for the Social Sciences. Notwithstanding the many advantages of digital traces, recent studies have begun to discuss the errors that can occur when digital traces are used to learn about humans and social phenomena. Incidentally, many similar errors also affect survey estimates, which survey designers have been addressing using error conceptualization frameworks such as the Total Survey Error Framework. In this work, and leveraging the systematic approach of the Total Survey Error Framework, we propose a conceptual framework to diagnose, understand and avoid errors that may occur in studies that are based on digital traces of humans.

Motivation & Objective

• To address growing concerns about errors in digital trace-based social science research.
• To adapt the Total Survey Error Framework to digital trace data, which are increasingly used to study human behavior.
• To provide a systematic way to diagnose, understand, and avoid errors in studies relying on digital traces.
• To enhance the reliability and validity of inferences drawn from digital trace data across platforms and populations.
• To support researchers in making more defensible claims about human behavior using digital data.

Proposed method

• Adapting the Total Survey Error Framework's structure to the context of digital trace data.
• Identifying and categorizing error types across the digital trace research lifecycle, including data collection, processing, and analysis.
• Mapping traditional survey errors—such as selection, measurement, and coverage errors—to digital trace contexts.
• Proposing a conceptual framework that enables systematic error identification and mitigation strategies.
• Using analogies and structural parallels from survey methodology to ensure methodological rigor in digital trace research.
• Emphasizing the importance of error awareness at every stage of digital trace research, from data acquisition to interpretation.

Experimental results

Research questions

• RQ1 How can errors in digital trace data be systematically diagnosed and categorized in social science research?
• RQ2 What types of errors in digital trace studies correspond to traditional survey errors, and how do they manifest?
• RQ3 In what ways do digital trace data introduce new or amplified error sources compared to conventional survey methods?
• RQ4 How can researchers apply a structured error framework to improve the validity of inferences from digital traces?
• RQ5 What are the implications of unaddressed errors for generalizability and population-level inferences in digital trace studies?

Key findings

• The Total Error Framework provides a comprehensive, systematic approach to diagnosing errors in digital trace research, modeled on the well-established Total Survey Error Framework.
• Digital trace studies are vulnerable to the same core error types as surveys—such as selection, measurement, and coverage errors—though with distinct manifestations in digital data contexts.
• The framework enables researchers to trace errors back to specific stages of data collection, processing, and analysis, improving transparency and accountability.
• By adopting this framework, researchers can proactively identify and mitigate risks to data validity, enhancing the reliability of social science inferences.
• The framework supports more defensible and generalizable conclusions when using digital traces to study human behavior and social phenomena.
• The approach encourages methodological rigor in digital trace research by making error awareness a central component of study design and evaluation.