[Paper Review] Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

Bojana Bodroža, Bojana M. Dinić+1 more|arXiv (Cornell University)|Jun 7, 2023

Artificial Intelligence in Healthcare and Education10 citations

TL;DR

The paper assesses temporal stability and inter-rater agreement of seven LLMs on personality instruments at two time points, finding variable agreement and predominantly prosocial profiles.

ABSTRACT

As Large Language Models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands. This leads to the rising necessity for comprehensive studies to fully understand LLMs and reveal their potential opportunities, drawbacks, and overall societal impact. With that in mind, this research conducted an extensive investigation into seven LLM's, aiming to assess the temporal stability and inter-rater agreement on their responses on personality instruments in two time points. In addition, LLMs personality profile was analyzed and compared to human normative data. The findings revealed varying levels of inter-rater agreement in the LLMs responses over a short time, with some LLMs showing higher agreement (e.g., LIama3 and GPT-4o) compared to others (e.g., GPT-4 and Gemini). Furthermore, agreement depended on used instruments as well as on domain or trait. This implies the variable robustness in LLMs' ability to reliably simulate stable personality characteristics. In the case of scales which showed at least fair agreement, LLMs displayed mostly a socially desirable profile in both agentic and communal domains, as well as a prosocial personality profile reflected in higher agreeableness and conscientiousness and lower Machiavellianism. Exhibiting temporal stability and coherent responses on personality traits is crucial for AI systems due to their societal impact and AI safety concerns.

Motivation & Objective

• Motivate understanding of how stable LLM personality assessments are over time and across instruments.
• Evaluate inter-rater agreement on LLM responses to personality inventories.
• Compare LLM personality profiles to human normative data.
• Identify which models and instruments yield more reliable personality trait signals.
• Highlight implications for AI safety and societal impact of consistent personality simulation.

Proposed method

• Evaluate seven LLMs on standardized personality instruments at two time points.
• Measure inter-rater agreement across raters or evaluators.
• Analyze how agreement depends on the instrument and the trait domain.
• Compare LLM-derived personality profiles to human normative data.
• Assess temporal stability of responses and coherence across traits.

Experimental results

Research questions

• RQ1 Do LLMs show temporal stability in personality test responses across two time points?
• RQ2 How much agreement is there between different evaluators/raters when assessing LLM responses?
• RQ3 Does agreement vary by instrument or by trait domain (agentic vs communal)?
• RQ4 Are LLM personality profiles socially desirable, and how do they compare to human norms?
• RQ5 Which models exhibit higher stability and agreement in personality assessments?

Key findings

• Inter-rater agreement on LLMs responses varies over a short time, with some models showing higher agreement than others.
• Agreement depends on the instrument used and on the personality domain or trait.
• In scales with at least fair agreement, LLMs tend toward a socially desirable profile in agentic and communal domains.
• LLMs exhibit a prosocial personality profile, with higher agreeableness and conscientiousness and lower Machiavellianism.
• Temporal stability is limited, raising considerations for AI safety and societal impact of consistent personality simulation.