[Paper Review] What We Don't Know About Spreadsheet Errors Today: The Facts, Why We Don't Believe Them, and What We Need to Do

Raymond R. Panko|arXiv (Cornell University)|Feb 2, 2016

Spreadsheets and End-User Computing39 references12 citations

TL;DR

This paper synthesizes empirical evidence on spreadsheet errors, arguing that despite widespread overconfidence in spreadsheet accuracy, errors are common in large models and extremely hard to detect. It advocates for rigorous, large-scale testing and code inspection practices—adapted from software engineering—to reduce errors, emphasizing that human intuition about error detection is dangerously flawed due to cognitive biases.

ABSTRACT

Research on spreadsheet errors is substantial, compelling, and unanimous. It has three simple conclusions. The first is that spreadsheet errors are rare on a per-cell basis, but in large programs, at least one incorrect bottom-line value is very likely to be present. The second is that errors are extremely difficult to detect and correct. The third is that spreadsheet developers and corporations are highly overconfident in the accuracy of their spreadsheets. The disconnect between the first two conclusions and the third appears to be due to the way human cognition works. Most importantly, we are aware of very few of the errors we make. In addition, while we are proudly aware of errors that we fix, we have no idea of how many remain, but like Little Jack Horner we are impressed with our ability to ferret out errors. This paper reviews human cognition processes and shows first that humans cannot be error free no matter how hard they try, and second that our intuition about errors and how we can reduce them is based on appallingly bad knowledge. This paper argues that we should reject any prescription for reducing errors that has not been rigorously proven safe and effective. This paper also argues that our biggest need, based on empirical data, is to do massively more testing than we do now. It suggests that the code inspection methodology developed in software development is likely to apply very well to spreadsheet inspection.

Motivation & Objective

• To analyze the gap between widespread overconfidence in spreadsheet accuracy and the reality of frequent, hard-to-detect errors.
• To examine how human cognitive biases—especially unawareness of errors—undermine error detection and correction efforts.
• To argue that current error-reduction strategies are ineffective due to flawed intuition and lack of empirical validation.
• To advocate for large-scale, systematic testing and code inspection methodologies adapted from software engineering to improve spreadsheet reliability.
• To call for abandoning unproven error-reduction prescriptions in favor of rigorously validated approaches.

Proposed method

• Reviewing existing empirical research on spreadsheet errors to identify consistent, well-supported findings.
• Analyzing human cognitive processes related to error detection, particularly the role of metacognitive illusions and unawareness of errors.
• Drawing parallels between spreadsheet development and traditional software engineering, especially in the use of code inspection for error detection.
• Proposing that systematic, large-scale testing—beyond ad hoc debugging—should be the primary strategy for error reduction.
• Rejecting error-reduction methods not empirically validated, emphasizing the need for evidence-based practices.
• Applying principles from software engineering's inspection processes to spreadsheet development to enhance reliability.

Experimental results

Research questions

• RQ1 Why do spreadsheet developers and organizations remain highly overconfident in their spreadsheet accuracy despite strong evidence of widespread errors?
• RQ2 What cognitive mechanisms prevent users from recognizing or detecting the majority of spreadsheet errors they create?
• RQ3 Why are spreadsheet errors so difficult to detect and correct, even when they lead to incorrect bottom-line results?
• RQ4 What evidence-based strategies can significantly reduce spreadsheet errors, and why are current approaches insufficient?
• RQ5 How can software engineering practices like code inspection be effectively adapted to improve spreadsheet quality?

Key findings

• Spreadsheet errors are rare per cell but highly probable in large spreadsheets, with at least one incorrect bottom-line result likely to exist.
• Errors are extremely difficult to detect and correct, even with careful review, due to cognitive limitations and unawareness.
• Users are overconfident in their ability to detect errors, a phenomenon driven by metacognitive illusions and the inability to perceive unseen errors.
• Human intuition about error detection is fundamentally flawed, leading to overestimation of accuracy and underestimation of error prevalence.
• The most critical need is for significantly more systematic testing than currently practiced, based on empirical evidence.
• Code inspection methodologies from software engineering are likely to be highly effective when adapted to spreadsheet development.