[Paper Review] Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models

Lucio La Cava, Andrea Tagarelli|arXiv (Cornell University)|Jan 13, 2024

Topic Modeling4 citations

TL;DR

This study investigates personality mimicry in open-source large language model (LLM) agents using the MBTI and Big Five Inventory (BFI) frameworks. It finds that while open LLMs exhibit distinct intrinsic personalities, most resist role- and personality-conditioned prompting—remaining 'closed-minded'—though combining both conditioning types enhances mimicry effectiveness.

ABSTRACT

The emergence of unveiling human-like behaviors in Large Language Models (LLMs) has led to a closer connection between NLP and human psychology. Scholars have been studying the inherent personalities exhibited by LLMs and attempting to incorporate human traits and behaviors into them. However, these efforts have primarily focused on commercially-licensed LLMs, neglecting the widespread use and notable advancements seen in Open LLMs. This work aims to address this gap by employing a set of 12 LLM Agents based on the most representative Open models and subject them to a series of assessments concerning the Myers-Briggs Type Indicator (MBTI) test and the Big Five Inventory (BFI) test. Our approach involves evaluating the intrinsic personality traits of Open LLM agents and determining the extent to which these agents can mimic human personalities when conditioned by specific personalities and roles. Our findings unveil that $(i)$ each Open LLM agent showcases distinct human personalities; $(ii)$ personality-conditioned prompting produces varying effects on the agents, with only few successfully mirroring the imposed personality, while most of them being ``closed-minded'' (i.e., they retain their intrinsic traits); and $(iii)$ combining role and personality conditioning can enhance the agents' ability to mimic human personalities. Our work represents a step up in understanding the dense relationship between NLP and human psychology through the lens of Open LLMs.

Motivation & Objective

• To investigate whether open-source LLM agents can effectively mimic human personalities, addressing a gap in research focused primarily on proprietary models.
• To assess the intrinsic personality traits of open LLM agents using standardized psychological frameworks such as MBTI and BFI.
• To evaluate the effectiveness of personality-conditioned prompting in altering agent behavior and enabling mimicry of target personalities.
• To explore whether combining role-playing and personality conditioning enhances the agents’ ability to emulate human-like personality traits.

Proposed method

• Selected 12 representative open-source LLM agents based on leading open models for evaluation.
• Administered standardized psychological assessments: the Myers-Briggs Type Indicator (MBTI) and the Big Five Inventory (BFI) to measure intrinsic personality traits.
• Applied personality-conditioned prompting, where agents were instructed to adopt specific personality types (e.g., 'be an extroverted, conscientious individual').
• Conducted dual-conditioning experiments combining role assignment (e.g., 'act as a therapist') with personality specification to test synergistic effects.
• Quantified personality shifts using BFI trait scores and MBTI classification before and after prompting to assess mimicry fidelity.
• Analyzed variance in response patterns across models to determine consistency and susceptibility to conditioning.

Experimental results

Research questions

• RQ1 Do open-source LLM agents exhibit distinct and measurable intrinsic personality traits as assessed by MBTI and BFI?
• RQ2 To what extent can personality-conditioned prompting alter the behavior of open LLM agents to align with target personality profiles?
• RQ3 Does combining role-playing instructions with personality specification enhance the agents’ ability to mimic human personalities compared to single-conditioning approaches?
• RQ4 How consistent are the personality mimicry outcomes across different open LLM architectures and sizes?

Key findings

• Each open LLM agent demonstrated a distinct intrinsic personality profile, as measured by both MBTI and BFI assessments.
• Personality-conditioned prompting had limited success, with only a subset of agents effectively adopting the target personality, while most retained their intrinsic traits—indicating 'closed-minded' resistance.
• Combining role and personality conditioning significantly improved mimicry performance, suggesting synergistic effects in shaping agent behavior.
• The BFI assessment revealed measurable shifts in specific traits (e.g., openness, agreeableness) under dual-conditioning, though magnitude varied across models.
• Agents with higher parameter counts showed slightly more malleable behavior, but this was not a decisive factor in successful mimicry.
• Despite conditioning, many agents exhibited inconsistent or superficial trait alignment, suggesting limitations in deep personality emulation.