[Paper Review] What Makes Agile Test Artifacts Useful? An Activity-Based Quality Model from a Practitioners' Perspective

Jannik Fischbach, Henning Femmer+3 more|arXiv (Cornell University)|Sep 3, 2020

Software Engineering Research4 citations

TL;DR

This study proposes an Activity-Based Artifact Quality Model (ABAQM) for Agile test artifacts, derived from industrial surveys with 18 practitioners across 12 companies. It identifies 16 concrete quality factors that make test artifacts—such as acceptance criteria, test documentation, and unit tests—more useful in practice, emphasizing language clarity, traceability, and automation to improve testing efficiency and quality.

ABSTRACT

Background: The artifacts used in Agile software testing and the reasons why these artifacts are used are fairly well-understood. However, empirical research on how Agile test artifacts are eventually designed in practice and which quality factors make them useful for software testing remains sparse. Aims: Our objective is two-fold. First, we identify current challenges in using test artifacts to understand why certain quality factors are considered good or bad. Second, we build an Activity-Based Artifact Quality Model that describes what Agile test artifacts should look like. Method: We conduct an industrial survey with 18 practitioners from 12 companies operating in seven different domains. Results: Our analysis reveals nine challenges and 16 factors describing the quality of six test artifacts from the perspective of Agile testers. Interestingly, we observed mostly challenges regarding language and traceability, which are well-known to occur in non-Agile projects. Conclusions: Although Agile software testing is becoming the norm, we still have little confidence about general do's and don'ts going beyond conventional wisdom. This study is the first to distill a list of quality factors deemed important to what can be considered as useful test artifacts.

Motivation & Objective

• To identify practical challenges Agile testers face when using test artifacts in real-world projects.
• To understand why certain quality factors are perceived as beneficial or detrimental in Agile testing workflows.
• To develop an empirically grounded, activity-based quality model that defines what high-quality test artifacts should look like from practitioners’ perspectives.
• To move beyond abstract normative standards by grounding quality criteria in actual testing activities and stakeholder needs.

Proposed method

• Conducted a qualitative industrial survey using one-on-one interviews with 18 experienced Agile practitioners from 12 companies across seven domains.
• Applied the Activity-Based Artifact Quality Model (ABAQM) framework to link artifact properties to stakeholder activities and quality outcomes.
• Used iterative questionnaire design with pilot testing and internal/external validation to ensure question relevance and clarity.
• Performed thematic analysis on interview transcripts to extract challenges and quality factors, followed by member checking for credibility.
• Mapped identified quality factors to specific test artifacts (e.g., acceptance criteria, unit tests) and linked them to stakeholder activities.
• Validated findings through internal review and participant feedback to reduce researcher bias and enhance construct validity.

Experimental results

Research questions

• RQ1 What challenges do Agile practitioners encounter when using test artifacts in their daily testing activities?
• RQ2 Which specific quality factors influence the usefulness of Agile test artifacts from the perspective of testers and other stakeholders?
• RQ3 How do the quality factors of test artifacts relate to the activities they support in Agile testing workflows?
• RQ4 To what extent do existing normative standards fail to capture the practical quality needs of Agile test artifacts?

Key findings

• Nine key challenges were identified, with 'lack of traceability' and 'ambiguous acceptance criteria' being the most frequently cited issues.
• The most critical quality factors include clarity of language, proper traceability between artifacts, and consistent version control across test assets.
• Practitioners reported that poorly documented test results and missing test data significantly reduce testing effectiveness.
• Automation of acceptance tests was frequently cited as a missing or under-implemented factor, leading to increased manual effort.
• Unit tests were often criticized for inadequate code coverage and lack of maintenance, undermining long-term test reliability.
• The study reveals that despite Agile principles favoring minimal documentation, teams still rely heavily on well-designed artifacts for quality assurance and team coordination.