[Paper Review] An Independent Validation of Vulnerability Discovery Models

Viet Hung Nguyen, Fabio Massacci|arXiv (Cornell University)|Mar 26, 2012

Software Reliability and Analysis Research11 references4 citations

TL;DR

This paper independently validates six vulnerability discovery models (VDMs) across 17 browser releases (Firefox, Chrome, IE), using five data sets defined by different vulnerability criteria. The AML (Alhazmi & Malaiya's Logistic) model shows superior performance, while the Anderson Thermodynamic (AT) model fails entirely, indicating that vulnerability discovery differs fundamentally from general software reliability modeling due to heightened motivation to find security flaws.

ABSTRACT

Having a precise vulnerability discovery model (VDM) would provide a useful quantitative insight to assess software security. Thus far, several models have been proposed with some evidence supporting their goodness-of-fit. In this work we describe an independent validation of the applicability of six existing VDMs in seventeen releases of the three popular browsers Firefox, Google Chrome and Internet Explorer. We have collected five different kinds of data sets based on different definitions of a vulnerability. We introduce two quantitative metrics, goodness-of-fit entropy and goodness-of-fit quality, to analyze the impact of vulnerability data sets to the stability as well as quality of VDMs in the software life cycles. The experiment result shows that the "confirmed-by-vendors' advisories" data sets apparently yields more stable and better results for VDMs. And the performance of the s-shape logistic model (AML) seems to be superior performance in overall. Meanwhile, Anderson thermodynamic model (AT) is indeed not suitable for modeling the vulnerability discovery process. This means that the discovery process of vulnerabilities and normal bugs are different because the interests of people in finding security vulnerabilities are more than finding normal programming bugs.

Motivation & Objective

• To assess the real-world applicability of six existing vulnerability discovery models (VDMs) in real software life cycles.
• To investigate how different definitions of vulnerabilities impact the stability and quality of VDM fits.
• To evaluate whether VDMs can reliably model the vulnerability discovery process in long-lived, evolving software like web browsers.
• To identify which VDMs are most suitable for guiding resource allocation in software security planning.

Proposed method

• The study collects five distinct data sets based on different vulnerability definitions: raw NVD entries, NVD entries confirmed by vendor advisories (NVD.Advice), and three variants derived from bug tracking systems (e.g., ADVICE.Nbug).
• The authors introduce two novel metrics: goodness-of-fit entropy and goodness-of-fit quality to quantitatively assess model stability and performance across data sets.
• Six VDMs are fitted to the data using nonlinear least-squares regression (via R's nls() function), with results cross-validated using CurveExpert Pro to ensure robustness.
• The analysis accounts for version-specific codebases by treating each browser release as an independent application, avoiding the error of aggregating across major version changes.
• Statistical tests are carefully selected to avoid assumptions of normality, given the non-normal distribution of vulnerability discovery events.
• The study uses a multi-release, multi-browser dataset spanning Firefox, Chrome, and Internet Explorer to ensure generalizability across different software development models.

Experimental results

Research questions

• RQ1 Which vulnerability discovery model performs best across multiple browser releases when validated independently?
• RQ2 How does the definition of a vulnerability (e.g., NVD-only vs. vendor-confirmed) affect the goodness-of-fit and stability of VDMs?
• RQ3 Can reliability-based models like the Anderson Thermodynamic (AT) model accurately describe the vulnerability discovery process in real-world software?
• RQ4 Do VDMs perform consistently across different browser versions, especially when major codebase changes occur?
• RQ5 To what extent does the choice of data set impact the perceived quality and reliability of VDM predictions?

Key findings

• The AML (Alhazmi & Malaiya's Logistic) model demonstrates the best overall performance in terms of goodness-of-fit quality and stability across data sets.
• The Anderson Thermodynamic (AT) model performs poorly in all cases, indicating it is unsuitable for modeling vulnerability discovery, likely due to its assumptions not matching real-world discovery behavior.
• The 'confirmed-by-vendor-advisories' data set yields significantly more stable and higher-quality model fits than raw NVD entries, suggesting vendor confirmation improves data reliability.
• The AML model's three-phase structure—learning, linear discovery, and saturation—best captures the actual dynamics of vulnerability discovery in practice.
• Long-lived software with code inheritance (e.g., Firefox) exhibit multiple saturation periods, which existing VDMs fail to model, explaining poor fits in such cases.
• Linear and exponential models (LN, RE, LP) perform comparably in releases where discovery follows a linear trend, but they lack the flexibility to model non-linear saturation phases.