[Paper Review] Secure Software Development Methodologies: A Multivocal Literature Review

Arina Kudriavtseva, Olga Gadyatskaya|arXiv (Cornell University)|Nov 29, 2022

Information and Cyber Security4 citations

TL;DR

This multivocal literature review analyzes 28 secure software development methodologies (SSDMs) from industry, government, and academia (2004–2022), mapping their technical and auxiliary (non-technical) security practices across the software development lifecycle (SDLC). It identifies critical research gaps, including weak evidence of effectiveness, limited inclusion of non-technical practices in academic SSDMs, and the lack of adoption of existing methodologies despite rising vulnerabilities.

ABSTRACT

In recent years, the number of cyber attacks has grown rapidly. An effective way to reduce the attack surface and protect software is adoption of methodologies that apply security at each step of the software development lifecycle. While different methodologies have been proposed to address software security, recent research shows an increase in the number of vulnerabilities in software and data breaches. Therefore, the security practices incorporated in secure software development methodologies require investigation. This paper provides an overview of security practices involved in 28 secure software development methodologies from industry, government, and academia. To achieve this goal, we distributed the security practices among the software development lifecycle stages. We also investigated auxiliary (non-technical) practices, such as organizational, behavioral, legal, policy, and governance aspects that are incorporated in the secure software development methodologies. Furthermore, we explored methods used to provide evidence of the effectiveness of the methodologies. Finally, we present the gaps that require attention in the scientific community. The results of our survey may assist researchers and organizations to better understand the existing security practices integrated into the secure software development methodologies. In addition, our bridge between "technical" and "non-technical" worlds may be useful for non-technical specialists who investigate software security. Moreover, exploring the gaps that we found in current research may help improve security in software development and produce software with fewer number of vulnerabilities.

Motivation & Objective

• To systematize security practices across 28 secure software development methodologies (SSDMs) from industry, government, and academia.
• To map these practices across the stages of the software development lifecycle (SDLC).
• To investigate auxiliary (non-technical) practices—such as governance, policy, culture, and communication—that support software security.
• To evaluate the evidence provided for the effectiveness of these methodologies.
• To identify research gaps in current SSDM research and practice.

Proposed method

• Conducted a multivocal literature review following Garousi et al. guidelines, integrating academic and industry sources.
• Collected 28 SSDMs issued between 2004 and 2022 from diverse sectors.
• Mapped security practices to SDLC phases (e.g., requirements, design, implementation, testing, deployment).
• Classified practices into technical (e.g., threat modeling, static analysis) and auxiliary (e.g., policy, culture, ethics, communication).
• Reviewed validation studies and evidence of effectiveness reported in each methodology.
• Identified research gaps through comparative analysis of content, scope, and evidence quality.

Experimental results

Research questions

• RQ1 What security practices are commonly integrated into secure software development methodologies across the SDLC?
• RQ2 How do secure software development methodologies incorporate non-technical (auxiliary) security practices such as governance, culture, and policy?
• RQ3 What types of evidence do methodologies provide to support their claimed effectiveness in reducing software vulnerabilities?
• RQ4 Why do organizations often develop custom SSDMs instead of adopting existing ones?
• RQ5 What are the key research gaps in current secure software development methodology research?

Key findings

• Most secure software development methodologies lack concrete evidence of effectiveness, with only one out of eight academic papers including a case study.
• Only two methodologies included case studies focused on requirements or design stages, indicating limited empirical validation.
• Academic SSDMs tend to emphasize technical practices while underrepresenting auxiliary practices such as security culture, policy, and team communication.
• Despite the availability of numerous SSDMs, the number of software vulnerabilities continues to rise, suggesting a gap between methodology development and real-world impact.
• Many methodologies do not clearly articulate what is novel about their approach, often reusing practices already documented in existing frameworks.
• There is a strong trend of organizations creating proprietary SSDMs rather than adopting established ones, possibly due to perceived lack of awareness or perceived implementation burden.