[Paper Review] Combining STPA and BDD for Safety Analysis and Verification in Agile Development: A Controlled Experiment

Yang Wang, Stefan Wagner|arXiv (Cornell University)|Apr 5, 2018

Software Engineering Techniques and Practices30 references3 citations

TL;DR

This paper proposes integrating Behavior-Driven Development (BDD) with System-Theoretic Process Analysis (STPA) for safety verification in agile development of safety-critical systems. The method uses BDD's scenario-based testing ('Given-When-Then') to verify safety requirements derived from STPA, showing significant improvement in communication effectiveness over traditional User Acceptance Testing (UAT), though no significant differences were found in productivity, test thoroughness, or fault detection.

ABSTRACT

Context: Agile development is in widespread use, even in safety-critical domains. Motivation: However, there is a lack of an ap- propriate safety analysis and verification method in agile development. Objective: In this paper, we investigate the use of Behavior Driven De- velopment (BDD) instead of standard User Acceptance Testing (UAT) for safety verification with System-Theoretic Process Analysis (STPA) for safety analysis in agile development. Method: We evaluate the effect of this combination in a controlled experiment with 44 students in terms of productivity, test thoroughness, fault detection effectiveness and com- munication effectiveness. Results: The results show that BDD is more effective for safety verification regarding the impact on communication effectiveness than standard UAT, whereas productivity, test thorough- ness and fault detection effectiveness show no statistically significant difference in our controlled experiment. Conclusion: The combination of BDD and STPA seems promising with an enhancement on communica- tion, but its impact needs more research.

Motivation & Objective

• Address the lack of integrated safety analysis and verification methods in agile development of safety-critical systems (SCS).
• Investigate whether BDD can serve as a more effective safety verification technique than traditional UAT in agile environments.
• Evaluate the impact of combining STPA for safety analysis with BDD for safety verification on key development metrics.
• Explore how BDD enhances communication around safety requirements during iterative development.
• Assess the feasibility and effectiveness of BDD in verifying safety-critical behaviors guided by system-theoretic principles.

Proposed method

• Apply STPA for safety analysis at the high-level architecture stage, deriving safety requirements without needing a detailed or stable system design.
• Use BDD's 'Given-When-Then' template to specify safety-critical test scenarios in plain English, focusing on system behavior rather than implementation.
• Implement BDD test cases in a test-last manner using Java code, linked to the system under development.
• Conduct a controlled experiment with 44 students, comparing BDD and UAT across four evaluation dimensions: productivity, test thoroughness, fault detection effectiveness, and communication effectiveness.
• Use automated test suites integrated with continuous integration to track test status and support real-time feedback.
• Collect and analyze data through surveys and performance metrics to assess the impact of BDD on communication and verification quality.

Experimental results

Research questions

• RQ1 Does using BDD instead of UAT improve communication effectiveness in safety verification within agile development?
• RQ2 Is there a statistically significant difference in productivity when using BDD versus UAT for safety verification?
• RQ3 How do BDD and UAT compare in terms of test thoroughness and fault detection effectiveness?
• RQ4 To what extent does BDD support early detection of functional gaps and edge cases in safety requirements?
• RQ5 How does BDD enhance bidirectional traceability and shared understanding between developers, business analysts, and safety stakeholders?

Key findings

• BDD significantly improves communication effectiveness compared to UAT, with 11 out of 24 communication aspects showing statistical significance.
• The 'Given-When-Then' format in BDD enhances clarity and reduces ambiguity, enabling better understanding and documentation of safety requirements.
• BDD test cases were perceived as better organized, more readable, and easier to maintain due to strict naming and hierarchical structure.
• Developers reported deeper engagement with safety requirements when writing BDD scenarios, leading to earlier identification of edge cases and functional gaps.
• Business analysts expressed higher confidence in BDD test cases due to their direct traceability to code and real-time verification status via automated reporting.
• No statistically significant differences were found between BDD and UAT in productivity, test thoroughness, or fault detection effectiveness, possibly due to the simplicity of the test code and manual mutation analysis.