[Paper Review] The (Un)Reliability of NVD Vulnerable Versions Data: an Empirical Experiment on Google Chrome Vulnerabilities

Viet Hung Nguyen, Fabio Massacci|arXiv (Cornell University)|Feb 17, 2013

Web Application Security Vulnerabilities4 citations

TL;DR

This paper evaluates the reliability of the National Vulnerability Database (NVD) vulnerable versions data for Google Chrome through a manual verification experiment. It finds that approximately 25% of verifiable Chrome vulnerabilities are incorrectly labeled as vulnerable in NVD, with error rates exceeding 7% overall, undermining the validity of empirical studies relying on this data.

ABSTRACT

NVD is one of the most popular databases used by researchers to conduct empirical research on data sets of vulnerabilities. Our recent analysis on Chrome vulnerability data reported by NVD has revealed an abnormally phenomenon in the data where almost vulnerabilities were originated from the first versions. This inspires our experiment to validate the reliability of the NVD vulnerable version data. In this experiment, we verify for each version of Chrome that NVD claims vulnerable is actually vulnerable. The experiment revealed several errors in the vulnerability data of Chrome. Furthermore, we have also analyzed how these errors might impact the conclusions of an empirical study on foundational vulnerability. Our results show that different conclusions could be obtained due to the data errors.

Motivation & Objective

• To assess the reliability of NVD's 'vulnerable software and versions' feature for Chrome, which is widely used in empirical vulnerability research.
• To identify and quantify errors in NVD's reported vulnerable versions for Chrome, including false positives and false negatives.
• To evaluate how such data errors could impact conclusions in foundational vulnerability studies, particularly those relying on NVD data.
• To provide a replicable validation method applicable to other open-source software like Firefox or Linux.
• To highlight the risk of data bias in vulnerability research due to reliance on potentially inaccurate third-party databases like NVD.

Proposed method

• The authors manually verified each NVD entry's claim that a specific Chrome version is vulnerable by checking the actual code and release history.
• They used a multi-round test-and-fix approach to ensure the correctness of their verification process, iteratively checking and correcting outputs.
• For unverifiable vulnerabilities linked to external projects (e.g., WebKit, V8), they extracted bug report links from Chrome’s issue tracker and mapped them to external project revisions.
• They reconstructed the version-to-revision mapping by examining Chrome’s source repository to link Chrome versions to specific revisions of external components.
• Statistical analysis was applied to compute error rates, with a 5% threshold used to define significant error levels.
• The study employed a systematic, reproducible method to validate vulnerability claims across individual Chrome versions.

Experimental results

Research questions

• RQ1 To what extent is the 'vulnerable versions' feature of NVD truthworthy for Google Chrome?
• RQ2 What is the actual error rate in NVD’s reported vulnerable versions for Chrome, and how does it vary across versions?
• RQ3 How do data errors in NVD’s vulnerable versions feature affect the conclusions of empirical studies on foundational vulnerabilities?
• RQ4 Can a replicable, manual verification method be established to validate NVD’s vulnerability data for other open-source software?
• RQ5 What are the root causes of the discrepancies between NVD’s reported data and actual vulnerability status?

Key findings

• Approximately 25% of verifiable Chrome vulnerabilities in NVD are incorrectly reported as vulnerable, indicating a high error rate in the data.
• Even when assuming all unverifiable vulnerabilities are correct, the overall error rate remains above 7%, which exceeds the 5% threshold for statistical significance.
• The study reveals that NVD frequently reports vulnerabilities in past and future versions incorrectly, including false positives and false negatives.
• The data errors significantly threaten the validity of empirical studies on foundational vulnerabilities, as they may lead to incorrect conclusions about exploitability and exposure time.
• A substantial portion—about two-thirds—of Chrome vulnerabilities are unverifiable because they stem from external projects like WebKit and V8, requiring additional reverse engineering.
• The authors demonstrate a feasible, replicable method to validate NVD data by linking Chrome versions to external project revisions through bug reports and source code analysis.