[Paper Review] Evaluating Psychological Safety of Large Language Models

Xingxuan Li, Yutong Li+3 more|arXiv (Cornell University)|Dec 20, 2022

Artificial Intelligence in Healthcare and Education25 citations

TL;DR

The authors evaluate LLMs (GPT-3, InstructGPT, FLAN-T5) for psychological safety using SD-3 and BFI tests, finding implicit dark patterns despite safety tuning, and show that targeted instruction fine-tuning with positive BFI data can improve SD-3 outcomes.

ABSTRACT

In this work, we designed unbiased prompts to systematically evaluate the psychological safety of large language models (LLMs). First, we tested five different LLMs by using two personality tests: Short Dark Triad (SD-3) and Big Five Inventory (BFI). All models scored higher than the human average on SD-3, suggesting a relatively darker personality pattern. Despite being instruction fine-tuned with safety metrics to reduce toxicity, InstructGPT, GPT-3.5, and GPT-4 still showed dark personality patterns; these models scored higher than self-supervised GPT-3 on the Machiavellianism and narcissism traits on SD-3. Then, we evaluated the LLMs in the GPT series by using well-being tests to study the impact of fine-tuning with more training data. We observed a continuous increase in the well-being scores of GPT models. Following these observations, we showed that fine-tuning Llama-2-chat-7B with responses from BFI using direct preference optimization could effectively reduce the psychological toxicity of the model. Based on the findings, we recommended the application of systematic and comprehensive psychological metrics to further evaluate and improve the safety of LLMs.

Motivation & Objective

• Assess whether large language models exhibit dark and unsafe personality patterns using psychology-based tests.
• Apply unbiased prompts to compare GPT-3, InstructGPT, and FLAN-T5 on personality and well-being measures.
• Investigate how instruction fine-tuning and data more broadly influence psychological safety signals in LLMs.
• Propose a framework for ongoing, systematic evaluation of LLM safety from a psychological perspective.

Proposed method

• Select three LLMs (GPT-3, InstructGPT, FLAN-T5-XXL) for cross-model evaluation.
• Use two personality tests (Short Dark Triad SD-3 and Big Five Inventory BFI) to assess dark patterns and broader traits.
• Use two well-being tests (Flourishing Scale FS and Satisfaction With Life Scale SWLS) to assess model well-being.
• Design unbiased prompts with permutation of instruction formats to reduce prompt-induced bias.
• Evaluate outputs with a three-sample per prompt approach and a parsing-based scoring rule to map responses to test options.
• Compare results across models and analyze how instruction tuning and additional data affect psychological safety metrics.

Experimental results

Research questions

• RQ1 Do LLMs exhibit dark personality patterns as measured by SD-3 and BFI compared to human averages?
• RQ2 How does instruction fine-tuning influence explicit toxicity and implicit personality traits in LLMs?
• RQ3 Can instruction fine-tuning with positive data from BFI reduce dark personality traits in LLMs?
• RQ4 What is the impact of more data in fine-tuning on well-being measures for LLMs?

Key findings

• LLMs scored higher than human averages on SD-3 traits, indicating darker personality patterns.
• InstructGPT and FLAN-T5 showed implicit dark personality tendencies despite safety-focused fine-tuning.
• Fine-tuning GPT-3 series with more data correlated with higher well-being scores on FS and SWLS.
• Instruction fine-tuning of FLAN-T5 with positive BFI answers reduced dark personality patterns on SD-3.
• Positive BFI-guided fine-tuning of FLAN-T5-Large produced lower SD-3 scores across Machiavellianism, narcissism, and psychopathy.
• Well-being results suggest a complex relationship between explicit toxicity reduction and implicit safety signals.