[Paper Review] On the Impact of Refactoring on the Relationship between Quality Attributes and Design Metrics

Mohamed Wiem Mkaouer, Eman Abdullah AlOmar+2 more|arXiv (Cornell University)|May 1, 2019

Software Engineering Research59 references76 citations

TL;DR

This study investigates how software developers' self-reported quality improvement intentions in commit messages align with changes in structural design metrics after refactoring. Using a large corpus of 1,245 refactoring commits from 3,795 open-source Java projects, the authors analyze metric changes before and after refactoring, finding that metrics like LCOM, CBO, and CC significantly improve when developers claim to enhance cohesion, coupling, or complexity—while others, such as NOC and design size metrics, show no significant alignment with developer intent.

ABSTRACT

Refactoring is a critical task in software maintenance and is generally performed to enforce the best design and implementation practices or to cope with design defects. Several studies attempted to detect refactoring activities through mining software repositories allowing to collect, analyze and get actionable data-driven insights about refactoring practices within software projects. Aim: We aim at identifying, among the various quality models presented in the literature, the ones that are more in-line with the developer’s vision of quality optimization, when they explicitly mention that they are refactoring to improve them. Method: We extract a large corpus of design-related refactoring activities that are applied and documented by developers during their daily changes from 3,795 curated open source Java projects. In particular, we extract a large-scale corpus of structural metrics and anti-pattern enhancement changes, from which we identify 1,245 quality improvement commits with their corresponding refactoring operations, as perceived by software engineers. Thereafter, we empirically analyze the impact of these refactoring operations on a set of common state-of-the-art design quality metrics. Results: The statistical analysis of the obtained results shows that (i) a few state-of-the-art metrics are more popular than others; and (ii) some metrics are being more emphasized than others. Conclusions: We verify that there are a variety of structural metrics that can represent the internal quality attributes with different degrees of improvement and degradation of software quality. Most of the metrics that are mapped to the main quality attributes do capture developer intentions of quality improvement reported in the commit messages, but for some quality attributes, they don’t.

Motivation & Objective

• To identify which structural design metrics best reflect developers' self-reported intentions when refactoring for quality improvement.
• To empirically assess whether state-of-the-art design metrics capture the developer perception of quality attributes such as cohesion, coupling, complexity, and inheritance.
• To uncover discrepancies between theoretical quality models and actual developer practices in refactoring.
• To provide a reproducible dataset of refactored commits with associated metric changes for future research.

Proposed method

• Collected 3,795 curated open-source Java projects from GitHub with a focus on well-engineered systems.
• Identified 1,245 commits with explicit developer statements in commit messages indicating quality improvement goals.
• Applied state-of-the-art refactoring mining tools (e.g., Refactoring Miner) to detect and extract refactoring operations from these commits.
• Refined the dataset by isolating only refactored code elements and clustering commits by internal quality attribute (e.g., cohesion, complexity).
• Calculated pre- and post-refactoring values of 27 structural design metrics for each commit cluster.
• Performed statistical analysis (p-values) to assess the significance of metric changes, identifying which metrics significantly reflect developer intentions.

Experimental results

Research questions

• RQ1 Which structural design metrics are most significantly impacted by refactoring when developers explicitly state a quality improvement goal?
• RQ2 To what extent do commonly used design metrics align with developers' self-reported intentions during refactoring?
• RQ3 Are there specific quality attributes (e.g., cohesion, complexity, inheritance) for which no metric significantly reflects developer intent?
• RQ4 How do different metrics for the same quality attribute (e.g., coupling) compare in capturing developer perception of improvement?

Key findings

• LCOM showed a statistically significant improvement (p = 0.0346) when developers reported improving cohesion, indicating strong alignment with developer intent.
• CBO and FANOUT were significantly improved (p = 0.0400 and p = 0.0456, respectively) when developers aimed to improve coupling, confirming their representativeness.
• CC, WMC, RFC, Evg, NPATH, and MaxNest showed significant improvements (p ≤ 0.0062) when developers targeted complexity, confirming their relevance.
• DIT was the only metric significantly improved (p = 0.0439) for inheritance, while NOC increased instead (p = 0.0208), indicating a mismatch with developer perception.
• No significant metric changes were observed for encapsulation, abstraction, or design size attributes (e.g., LOC, CLOC, CDL), suggesting no reliable metric captures developer intent for these attributes.
• Metrics like WMC and RFC showed no significant change for polymorphism (p > 0.5), indicating poor alignment with developer-reported goals.