[Paper Review] Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

Wentao Ye, Mingfeng Ou+9 more|arXiv (Cornell University)|May 15, 2023

Topic Modeling13 citations

TL;DR

This paper presents an automated workflow to study LLMs’ robustness, consistency, and credibility using over one million queries across multiple models, revealing vulnerabilities and proposing the Relative Training Index (RTI) to gauge dataset credibility for LLM evaluations.

ABSTRACT

The recent popularity of large language models (LLMs) has brought a significant impact to boundless fields, particularly through their open-ended ecosystem such as the APIs, open-sourced models, and plugins. However, with their widespread deployment, there is a general lack of research that thoroughly discusses and analyzes the potential risks concealed. In that case, we intend to conduct a preliminary but pioneering study covering the robustness, consistency, and credibility of LLMs systems. With most of the related literature in the era of LLM uncharted, we propose an automated workflow that copes with an upscaled number of queries/responses. Overall, we conduct over a million queries to the mainstream LLMs including ChatGPT, LLaMA, and OPT. Core to our workflow consists of a data primitive, followed by an automated interpreter that evaluates these LLMs under different adversarial metrical systems. As a result, we draw several, and perhaps unfortunate, conclusions that are quite uncommon from this trendy community. Briefly, they are: (i)-the minor but inevitable error occurrence in the user-generated query input may, by chance, cause the LLM to respond unexpectedly; (ii)-LLMs possess poor consistency when processing semantically similar query input. In addition, as a side finding, we find that ChatGPT is still capable to yield the correct answer even when the input is polluted at an extreme level. While this phenomenon demonstrates the powerful memorization of the LLMs, it raises serious concerns about using such data for LLM-involved evaluation in academic development. To deal with it, we propose a novel index associated with a dataset that roughly decides the feasibility of using such data for LLM-involved evaluation. Extensive empirical studies are tagged to support the aforementioned claims.

Motivation & Objective

• Motivate systematic evaluation of LLM risks beyond traditional NLP metrics.
• Propose an automated workflow to scale robustness, consistency, and credibility assessment for LLMs.
• Introduce a unified data primitive and automated interpreter to handle large query responses.
• Develop threat models and attack schemes tailored to real-world LLM usage.
• Introduce RTI as a dataset credibility index to guide dataset selection for LLM evaluation.

Proposed method

• Use the gpt-3.5-turbo API, and open-source LLaMA and OPT models as backbones.
• Form a universal data primitive: (prompt, p, q, o, a) to structure QA data with multiple confusing options.
• Automate attacks via word/character/visual perturbations to simulate realistic input errors.
• Define robustness and consistency threat models and five attack schemes aligned with LLM usage scenarios.
• Compute RTI by progressively perturbing inputs to determine memorization effects and dataset reliability.
• Open-source datasets and samples are provided at the project URL.

Experimental results

Research questions

• RQ1 How robust are leading LLMs to adversarially structured inputs and common user errors?
• RQ2 How consistent are LLM responses when semantically similar inputs are reformulated?
• RQ3 Can we quantify the credibility of evaluation datasets for LLM-based assessment using a memorization-driven index (RTI)?
• RQ4 What practical implications do robustness, consistency, and memorization have for academic evaluation of LLMs?

Key findings

• Minor input perturbations can cause LLMs to respond unexpectedly.
• LLMs show poor consistency when processing semantically similar queries.
• ChatGPT can still yield correct answers even when input is heavily polluted, indicating memorization.
• RTI provides a relative measure of dataset memorization and suitability for LLM evaluation.
• The study emphasizes caution in using polluted or memorized datasets for LLM-involved evaluation and provides open-source resources.