[Paper Review] Large Language Models Understand and Can be Enhanced by Emotional Stimuli

Cheng Li, Jindong Wang+7 more|arXiv (Cornell University)|Jul 14, 2023

Topic Modeling57 citations

TL;DR

The paper introduces EmotionPrompt, which appends psychological emotional stimuli to prompts and shows LLMs understand and improve with these cues across deterministic and generative tasks, validated by both automatic benchmarks and a human study.

ABSTRACT

Emotional intelligence significantly impacts our daily behaviors and interactions. Although Large Language Models (LLMs) are increasingly viewed as a stride toward artificial general intelligence, exhibiting impressive performance in numerous tasks, it is still uncertain if LLMs can genuinely grasp psychological emotional stimuli. Understanding and responding to emotional cues gives humans a distinct advantage in problem-solving. In this paper, we take the first step towards exploring the ability of LLMs to understand emotional stimuli. To this end, we first conduct automatic experiments on 45 tasks using various LLMs, including Flan-T5-Large, Vicuna, Llama 2, BLOOM, ChatGPT, and GPT-4. Our tasks span deterministic and generative applications that represent comprehensive evaluation scenarios. Our automatic experiments show that LLMs have a grasp of emotional intelligence, and their performance can be improved with emotional prompts (which we call "EmotionPrompt" that combines the original prompt with emotional stimuli), e.g., 8.00% relative performance improvement in Instruction Induction and 115% in BIG-Bench. In addition to those deterministic tasks that can be automatically evaluated using existing metrics, we conducted a human study with 106 participants to assess the quality of generative tasks using both vanilla and emotional prompts. Our human study results demonstrate that EmotionPrompt significantly boosts the performance of generative tasks (10.9% average improvement in terms of performance, truthfulness, and responsibility metrics). We provide an in-depth discussion regarding why EmotionPrompt works for LLMs and the factors that may influence its performance. We posit that EmotionPrompt heralds a novel avenue for exploring interdisciplinary knowledge for human-LLMs interaction.

Motivation & Objective

• Assess whether LLMs can understand psychological emotional stimuli.
• Develop and test EmotionPrompt by adding emotional cues to prompts.
• Evaluate EmotionPrompt on deterministic tasks with automatic metrics.
• Evaluate EmotionPrompt on generative tasks with human judgments.

Proposed method

• Design 11 emotional stimuli based on Self-monitoring, Social Cognitive Theory, and Cognitive Emotion Regulation Theory as prompt appendages (EmotionPrompt).
• Evaluate zero-shot and few-shot LLM performance across six models (Flan-T5-Large, Vicuna, Llama 2, BLOOM, ChatGPT, GPT-4) on Instruction Induction and BIG-Bench."
• Use automatic metrics for Instruction Induction (accuracy) and BIG-Bench (normalized preferred score).
• Conduct a human study with 106 participants judging GPT-4 outputs on performance, truthfulness, and responsibility.
• Compare EmotionPrompt with baselines including Original prompts, Zero-shot-CoT, and APE.
• Perform ablation and analysis to understand which stimuli work best and why.

Figure 1 : An overview of our research from generating to evaluating EmotionPrompt.

Experimental results

Research questions

• RQ1 Can LLMs understand and benefit from emotional stimuli appended to prompts?
• RQ2 How much can EmotionPrompt improve deterministic task performance across diverse LLMs?
• RQ3 Does EmotionPrompt enhance generative task quality, truthfulness, and responsibility as judged by humans?

Key findings

• EmotionPrompt yields 8.00% relative improvement on Instruction Induction and 115% on BIG-Bench across evaluated models.
• In a human study with 106 participants, EmotionPrompt achieves 10.9% average improvement in performance, truthfulness, and responsibility for generative tasks.
• TruthfulQA improvements with EmotionPrompt average 19% in truthfulness and 12% in informativeness across models (best results shown with certain EP prompts).
• Ablation analyses show performance gains linked to emotional stimuli influencing final outputs via input attention and gradient effects, with certain prompts (EP02, EP06) performing best in respective benchmarks.
• EmotionPrompt remains generally effective across model sizes, task types, and other prompt engineering baselines, though not universally superior in all cases.

Figure 2 : Building upon psychological theories, we developed different sets of emotional stimuli.