[Paper Review] Beyond ChatGPT: Enhancing Software Quality Assurance Tasks with Diverse LLMs and Validation Techniques

Ratnadira Widyasari, David Lo+1 more|arXiv (Cornell University)|Sep 2, 2024

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

This paper evaluates diverse LLMs—GPT-3.5, GPT-4o, LLaMA-3-70B, LLaMA-3-8B, Gemma-7B, and Mixtral-8x7B—for fault localization and vulnerability detection in software quality assurance. It proposes a voting mechanism and cross-validation using validation prompts, achieving over 10% improvement over GPT-3.5 and up to 16% improvement in fault localization compared to the best single LLM.

ABSTRACT

With the advancement of Large Language Models (LLMs), their application in Software Quality Assurance (SQA) has increased. However, the current focus of these applications is predominantly on ChatGPT. There remains a gap in understanding the performance of various LLMs in this critical domain. This paper aims to address this gap by conducting a comprehensive investigation into the capabilities of several LLMs across two SQA tasks: fault localization and vulnerability detection. We conducted comparative studies using GPT-3.5, GPT-4o, and four other publicly available LLMs (LLaMA-3-70B, LLaMA-3-8B, Gemma-7B, and Mixtral-8x7B), to evaluate their effectiveness in these tasks. Our findings reveal that several LLMs can outperform GPT-3.5 in both tasks. Additionally, even the lower-performing LLMs provided unique correct predictions, suggesting the potential of combining different LLMs' results to enhance overall performance. By implementing a voting mechanism to combine the LLMs' results, we achieved more than a 10% improvement over the GPT-3.5 in both tasks. Furthermore, we introduced a cross-validation approach to refine the LLM answer by validating one LLM answer against another using a validation prompt. This approach led to performance improvements of 16% in fault localization and 12% in vulnerability detection compared to the GPT-3.5, with a 4% improvement compared to the best-performed LLMs. Our analysis also indicates that the inclusion of explanations in the LLMs' results affects the effectiveness of the cross-validation technique.

Motivation & Objective

• To investigate the performance of diverse LLMs beyond GPT-3.5 in critical SQA tasks like fault localization and vulnerability detection.
• To identify performance variations across LLMs based on model size, architecture, and task complexity.
• To explore ensemble techniques—voting and cross-validation using validation prompts—to enhance LLM-based SQA outcomes.
• To assess the impact of explanation quality on the effectiveness of cross-validation refinement.
• To provide a benchmark for selecting and combining LLMs in SQA workflows, moving beyond reliance on a single model like GPT-3.5.

Proposed method

• Compared six LLMs—GPT-3.5, GPT-4o, LLaMA-3-70B, LLaMA-3-8B, Gemma-7B, and Mixtral-8x7B—on two SQA tasks using standardized datasets from prior studies.
• Applied a voting mechanism where the final prediction across LLMs was determined by majority consensus to combine complementary strengths.
• Designed a cross-validation approach where one LLM’s output was refined by validating it against another LLM’s response using a dedicated validation prompt.
• Used GPT-4o and LLaMA-3-70B as the primary refinement pair in fault localization, and GPT-4o and Gemma-7B in vulnerability detection.
• Evaluated performance using standard metrics: Top-1 accuracy for fault localization and F1-score for vulnerability detection.
• Conducted three independent runs per LLM to ensure result stability and reduce randomness in evaluation.

Figure 1. Illustration of the proposed approach on cross-validating GPT-4o and LLaMA-3-70B

Experimental results

Research questions

• RQ1 How do different LLMs perform on fault localization and vulnerability detection compared to GPT-3.5, the current baseline in SQA research?
• RQ2 Can combining predictions from multiple LLMs via voting improve SQA task performance beyond individual models?
• RQ3 Does cross-validation using a validation prompt significantly enhance the accuracy of LLM predictions in SQA tasks?
• RQ4 How does the inclusion of explanations in LLM outputs affect the effectiveness of the cross-validation refinement process?
• RQ5 Are smaller LLMs like Gemma-7B competitive in certain SQA tasks despite their reduced parameter count?

Key findings

• GPT-4o achieved the highest performance in fault localization, improving Top-1 accuracy by 16.24% over GPT-3.5.
• Gemma-7B outperformed GPT-3.5 in vulnerability detection by 7.8% in accuracy, despite being the smallest model in the study.
• The voting mechanism improved fault localization by 13.7% and vulnerability detection by 11.2% compared to GPT-3.5.
• Cross-validation using GPT-4o refined by LLaMA-3-70B improved fault localization performance by 16% over GPT-3.5 and 4.1% over LLaMA-3-70B alone.
• In vulnerability detection, GPT-4o refined by Gemma-7B achieved a 12% improvement over GPT-3.5 and a 7.1% improvement over GPT-4o alone.
• The effectiveness of cross-validation was influenced by the quality of explanations in LLM outputs, with models like Gemma-7B showing limited refinement capability.

Figure 2. Validation prompt