[Paper Review] Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways

Shubham Atreja, Joshua Ashkinaze+3 more|arXiv (Cornell University)|Jun 17, 2024

Computational and Text Analysis Methods4 citations

TL;DR

This study investigates how prompt design affects large language model (LLM) compliance and accuracy in computational social science tasks, testing four prompt features—definition inclusion, output type, explanation, and length—across three LLMs (ChatGPT, PaLM2, Falcon7b) on four tasks (toxicity, sentiment, rumor stance, news frames). Results show prompt design drastically influences both compliance and accuracy in unpredictable, task- and model-specific ways, with numerical scoring reducing performance and explanations altering label distributions.

ABSTRACT

Manually annotating data for computational social science tasks can be costly, time-consuming, and emotionally draining. While recent work suggests that LLMs can perform such annotation tasks in zero-shot settings, little is known about how prompt design impacts LLMs' compliance and accuracy. We conduct a large-scale multi-prompt experiment to test how model selection (ChatGPT, PaLM2, and Falcon7b) and prompt design features (definition inclusion, output type, explanation, and prompt length) impact the compliance and accuracy of LLM-generated annotations on four CSS tasks (toxicity, sentiment, rumor stance, and news frames). Our results show that LLM compliance and accuracy are highly prompt-dependent. For instance, prompting for numerical scores instead of labels reduces all LLMs' compliance and accuracy. The overall best prompting setup is task-dependent, and minor prompt changes can cause large changes in the distribution of generated labels. By showing that prompt design significantly impacts the quality and distribution of LLM-generated annotations, this work serves as both a warning and practical guide for researchers and practitioners.

Motivation & Objective

• To investigate how prompt design features affect LLM compliance and accuracy in computational social science annotation tasks.
• To address the lack of systematic, large-scale empirical evidence on prompt effectiveness across diverse LLMs and tasks.
• To examine whether concise, definition-inclusive, or explanation-requesting prompts improve or degrade performance.
• To highlight risks of inconsistent annotation distributions due to subtle prompt changes, which may bias social science research.
• To provide evidence-based best practices for researchers using LLMs in zero-shot annotation tasks.

Proposed method

• Conducted a large-scale, complete factorial experiment with 16 prompt variations (2×2×2×2) across four prompt dimensions: definition inclusion, output type (label vs. score), explanation request, and prompt length (standard vs. concise).
• Applied these prompts to three LLMs—ChatGPT, PaLM2, and Falcon7b—on four computational social science tasks: toxicity, sentiment, rumor stance, and news frame detection.
• Used a combined dataset of 362,928 LLM-generated annotations across all task-model-prompt combinations.
• Measured compliance (adherence to prompt instructions) and accuracy (agreement with human-annotated gold standards).
• Analyzed label distribution shifts across prompts, especially when explanations or numerical scoring were included.
• Reported quantitative results on compliance variance (e.g., up to 55% for Falcon7b on rumor stance) and accuracy variance (e.g., up to 14% for ChatGPT on news frames).

Experimental results

Research questions

• RQ1 How does including definitions in prompts affect LLM compliance and accuracy across different computational social science tasks?
• RQ2 What is the impact of requesting numerical scores instead of categorical labels on LLM compliance and accuracy?
• RQ3 How do concise prompts compare to standard-length prompts in terms of compliance and accuracy across models and tasks?
• RQ4 To what extent does requesting an explanation of the reasoning alter label distributions and compliance rates?
• RQ5 How do the effects of prompt design features vary across different LLMs (e.g., ChatGPT, PaLM2, Falcon7b) and tasks?

Key findings

• Prompting for numerical scores instead of labels significantly reduced compliance and accuracy across all LLMs and most tasks.
• Including definitions improved ChatGPT’s accuracy without reducing compliance, but reduced compliance for PaLM2 and Falcon7b.
• Concise prompts improved cost efficiency for PaLM2 on rumor stance without harming compliance or accuracy, but harmed performance in most other cases.
• Requesting explanations increased compliance but altered label distributions—e.g., ChatGPT labeled 34% more content as 'neutral' when explanations were required.
• The best-performing prompt configuration was highly task- and model-dependent, with minor changes causing large shifts in output distributions.
• Compliance varied by up to 55% across prompts for Falcon7b on rumor stance, and accuracy varied by up to 14% for ChatGPT on news frames, underscoring the unpredictability of prompt effects.