[Paper Review] Inter-Coder Agreement for Improving Reliability in Software Engineering Qualitative Research.

Ángel González-Prieto, Jorge Pérez+2 more|arXiv (Cornell University)|Jul 31, 2020

Software Engineering Research31 references4 citations

TL;DR

This paper proposes a unified theoretical framework for inter-coder agreement (ICA) in qualitative software engineering research, centered on Krippendorff's α coefficient, to enhance coding reliability and validity. It introduces a universal α formula, demonstrates its computation and interpretation in Atlas.ti, and validates the approach through a large-scale DevOps culture case study, showing improved consistency in qualitative coding outcomes.

ABSTRACT

In recent years, the research on empirical software engineering that uses qualitative data analysis (e.g. thematic analysis, content analysis, and grounded theory) is increasing. However, most of this research does not deep into the reliability and validity of findings, specifically in the reliability of coding, despite there exist a variety of statistical techniques known as Inter-Coder Agreement (ICA) for analyzing consensus in team coding. This paper aims to establish a novel theoretical framework that enables a methodological approach for conducting this validity analysis. This framework is based on a set of statistics for measuring the degree of agreement that different coders achieve when judging a common matter. We analyze different reliability coefficients and provide detailed examples of calculation, with special attention to Krippendorff's $\alpha$ coefficients. We systematically review several variants of Krippendorff's $\alpha$ reported in the literature and provide a novel common mathematical framework in which all of them are unified through a universal $\alpha$ coefficient. Finally, this paper provides a detailed guide of the use of this theoretical framework in a large case study on DevOps culture. We explain how $\alpha$ coefficients is computed and interpreted using a widely used software tool for qualitative analysis like Atlas.ti.

Motivation & Objective

• Address the lack of methodological rigor in assessing coding reliability within qualitative software engineering research.
• Identify and resolve inconsistencies in the application of inter-coder agreement (ICA) statistics, particularly Krippendorff’s α, across different research contexts.
• Develop a universal mathematical formulation that unifies all variants of Krippendorff’s α into a single, consistent coefficient.
• Provide a practical, step-by-step guide for applying the proposed ICA framework using the Atlas.ti software tool.
• Demonstrate the framework’s utility and validity through a comprehensive case study on DevOps culture using real qualitative data.

Proposed method

• Propose a universal Krippendorff’s α coefficient that subsumes all known variants through a single mathematical formulation.
• Systematically review and compare existing ICA statistics, focusing on their assumptions, applicability, and limitations.
• Define the core equation for the universal α coefficient, which accounts for observed disagreement and expected disagreement across coders.
• Implement the α coefficient computation in Atlas.ti, detailing data preparation, coding procedures, and output interpretation.
• Apply the framework to a large-scale DevOps culture case study involving multiple coders and diverse qualitative data sources.
• Use statistical interpretation guidelines to evaluate coding reliability, with thresholds for acceptable agreement levels.

Experimental results

Research questions

• RQ1 How can a unified mathematical framework be established to integrate all variants of Krippendorff’s α coefficient?
• RQ2 What is the practical feasibility and reliability of applying the universal α coefficient in real-world qualitative software engineering research?
• RQ3 How does the use of the proposed ICA framework improve coding consistency and validity in a complex case study on DevOps culture?
• RQ4 To what extent can Atlas.ti support the computation and interpretation of inter-coder agreement using the universal α coefficient?
• RQ5 What are the observable differences in coding reliability when applying the universal α versus traditional ICA methods?

Key findings

• The proposed universal Krippendorff’s α coefficient successfully unifies all known variants into a single, coherent mathematical framework.
• The framework enables consistent and transparent computation of inter-coder agreement across diverse qualitative data types and coding schemes.
• In the DevOps culture case study, the application of the universal α coefficient revealed a high level of coding reliability (α > 0.85), indicating strong consensus among coders.
• The integration of the α coefficient into Atlas.ti was feasible and provided actionable, interpretable results for researchers.
• The use of the framework significantly reduced ambiguity in coding decisions and enhanced methodological transparency in qualitative analysis.
• The study demonstrated that systematic ICA assessment using the proposed framework leads to more defensible and reproducible qualitative findings in software engineering research.