[Paper Review] Quantifying Technical Debt: A Systematic Mapping Study and a Conceptual Model

Judith Perera, Ewan Tempero+2 more|arXiv (Cornell University)|Mar 12, 2023

Software Engineering Research4 citations

TL;DR

This paper proposes a conceptual model, the Technical Debt Quantification Model (TDQM), to unify and standardize the diverse approaches to quantifying technical debt (TD) in software development. By mapping 39 TD quantification approaches through a common framework—the TDQM Approach Comparison Matrix—it enables systematic comparison, reveals key concepts like 'Interest' and 'Cost of Refactoring', and identifies underexplored areas such as 'New Code Cost' and 'Rework Cost', thereby supporting more coherent technical debt management decisions.

ABSTRACT

To effectively manage Technical Debt (TD), we need reliable means to quantify it. We conducted a Systematic Mapping Study (SMS) where we identified TD quantification approaches that focus on different aspects of TD. Some approaches base the quantification on the identification of smells, some quantify the Return on Investment (ROI) of refactoring, some compare an ideal state with the current state of a software in terms of the software quality, and some compare alternative development paths to reduce TD. It is unclear if these approaches are quantifying the same thing and if they support similar or different decisions regarding TD Management (TDM). This creates the problem of not being able to effectively compare and evaluate approaches. To solve this problem, we developed a novel conceptual model, the Technical Debt Quantification Model (TDQM), that captures the important concepts related to TD quantification and illustrates the relationships between them. TDQM can represent varied TD quantification approaches via a common uniform representation, the TDQM Approach Comparison Matrix, that allows performing useful comparisons and evaluations between approaches. This paper reports on the mapping study, the development of TDQM, and on applying TDQM to compare and evaluate TD quantification approaches.

Motivation & Objective

• To identify and analyze existing approaches to quantifying technical debt (TD) in software development.
• To address the lack of consensus and comparability among TD quantification methods, which hinders effective technical debt management (TDM).
• To develop a unified conceptual model that captures core concepts of TD quantification and their relationships.
• To create a standardized comparison framework—the TDQM Approach Comparison Matrix—for evaluating and contrasting diverse TD quantification approaches.
• To extend the model's applicability to non-code-related TD types such as Requirements Debt in future work.

Proposed method

• Conducting a Systematic Mapping Study (SMS) to identify and categorize 39 TD quantification approaches from the literature.
• Developing the Technical Debt Quantification Model (TDQM) by synthesizing key concepts from existing literature, focusing on code-related TD types (e.g., code, design, architectural debt).
• Defining core TDQM concepts: TD Item, Interest, Cost of Refactoring, New Code, Rework, and their inter-relationships.
• Creating the TDQM Approach Comparison Matrix as a uniform representation to map and compare different quantification approaches across the defined concepts.
• Applying TDQM to evaluate 39 approaches—33 pre-model and 6 post-model—to assess consistency, coverage, and insights.
• Using the model to identify trends, such as high frequency of mappings to 'Interest' and 'Cost of Refactoring', and to highlight under-researched areas like 'New Code Cost'.

Experimental results

Research questions

• RQ1 RQ1: What approaches to technical debt quantification have been proposed in the research literature?
• RQ2 RQ2: How can we model technical debt quantification in a way that captures its essential concepts and relationships?
• RQ3 RQ3: How can we compare and evaluate different technical debt quantification approaches using a unified framework?
• RQ4 RQ4: Which TDQM concepts are most frequently addressed in existing quantification approaches?
• RQ5 RQ5: What are the under-explored or under-represented aspects of technical debt quantification that the model reveals?

Key findings

• The TDQM Approach Comparison Matrix successfully enabled systematic and efficient comparison of 39 diverse TD quantification approaches, despite differences in form and focus.
• The concepts 'Interest' and 'Cost of Refactoring' received the highest number of direct mappings (22 each), indicating their central role in current quantification efforts.
• The concept 'TD Item' was directly mapped in 16 approaches, showing its foundational role in TD representation.
• The model revealed that 'New Code Cost' and 'Rework Cost'—constituents of TD Interest—are underexplored, highlighting a gap in current research.
• Architecture Debt was the type of TD most comprehensively covered across the TDQM concepts, suggesting a focus on architectural-level debt in existing work.
• The years 2012, 2015, and 2016 were the most active for TD quantification research, indicating a peak in conceptual development during that period.