[Paper Review] What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering

Federico Errica, Giuseppe Siracusano+2 more|arXiv (Cornell University)|Jun 18, 2024

Higher Education Learning Practices4 citations

TL;DR

This paper introduces two novel metrics—sensitivity and consistency—to evaluate how reliably large language models (LLMs) perform under prompt variations in text classification tasks. Sensitivity measures prediction shifts across semantically equivalent prompt rephrasings without requiring ground truth, while consistency evaluates label stability within the same class. The key contribution is a diagnostic framework that identifies LLM failure modes and guides prompt engineering for robustness, especially in production systems with multiple LLM stages.

ABSTRACT

Large Language Models (LLMs) changed the way we design and interact with software systems. Their ability to process and extract information from text has drastically improved productivity in a number of routine tasks. Developers that want to include these models in their software stack, however, face a dreadful challenge: debugging LLMs' inconsistent behavior across minor variations of the prompt. We therefore introduce two metrics for classification tasks, namely sensitivity and consistency, which are complementary to task performance. First, sensitivity measures changes of predictions across rephrasings of the prompt, and does not require access to ground truth labels. Instead, consistency measures how predictions vary across rephrasings for elements of the same class. We perform an empirical comparison of these metrics on text classification tasks, using them as guideline for understanding failure modes of the LLM. Our hope is that sensitivity and consistency will be helpful to guide prompt engineering and obtain LLMs that balance robustness with performance.

Motivation & Objective

• To address the challenge of inconsistent LLM behavior due to minor prompt variations, which hinders deployment in production environments.
• To develop diagnostic metrics that complement accuracy and do not rely on ground truth labels for evaluating LLM robustness.
• To guide prompt engineering by identifying failure modes such as over-sensitivity or low consistency across classes.
• To provide a framework for assessing LLM reliability in multi-stage software systems where prompt variations are common.

Proposed method

• Propose sensitivity as a metric measuring prediction variance across Q rephrasings of the same prompt, computed without ground truth labels.
• Define consistency as the average similarity of predictions across examples of the same class, using cosine similarity between prediction distributions.
• Use a diverse set of prompting strategies (e.g., few-shot, zero-shot, detailed descriptions) to evaluate LLMs on multiple text classification datasets (TREC, DBPedia, WoS, CB).
• Apply the metrics to Llama3 and GPT-3.5, analyzing both aggregate and class-wise behavior to detect failure patterns.
• Visualize results using histograms and violin plots to compare consistency distributions across prompting techniques and datasets.
• Use qualitative analysis to trace misclassifications to specific prompt design choices, such as label description wording or ordering.

Experimental results

Research questions

• RQ1 How does LLM prediction vary across semantically equivalent prompt rephrasings, and can this be quantified without ground truth?
• RQ2 To what extent do different prompting strategies affect the consistency of LLM predictions within the same class?
• RQ3 Can sensitivity and consistency metrics identify failure modes that are not captured by accuracy alone?
• RQ4 How do label description wording and ordering influence LLM robustness, and can these be diagnosed systematically?

Key findings

• Sensitivity effectively identifies prompt variations that drastically alter LLM predictions, even when accuracy remains high, revealing hidden fragility in model behavior.
• Consistency values were often below 0.75 across classes, indicating that LLMs frequently misclassify similar examples differently, especially for confusing classes like 'Description' and 'Entity'.
• The 1-shot prompting strategy reduced medium-level consistency and increased the number of pairs with zero consistency, contradicting the expectation that few-shot improves stability.
• Label description wording—such as adding an article like 'an Entity'—can significantly alter predictions, demonstrating high sensitivity to minor prompt changes.
• Qualitative analysis revealed that class confusion between 'Description' and 'Entity' was common, and tuning prompt descriptions for these classes improved consistency.
• The metrics uncovered non-intuitive behaviors, such as increased sensitivity when using detailed class descriptions, highlighting the need for systematic prompt diagnostics beyond mean performance.