[论文解读] Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

Bojana Bodroža, Bojana M. Dinić+1 more|arXiv (Cornell University)|Jun 7, 2023

Artificial Intelligence in Healthcare and Education被引用 10

一句话总结

本文在两个时间点对七个大模型在性格量表上的时间稳定性和评估者之间的一致性进行了评估，发现一致性存在差异，且多为亲社会型人格特征。

ABSTRACT

As Large Language Models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands. This leads to the rising necessity for comprehensive studies to fully understand LLMs and reveal their potential opportunities, drawbacks, and overall societal impact. With that in mind, this research conducted an extensive investigation into seven LLM's, aiming to assess the temporal stability and inter-rater agreement on their responses on personality instruments in two time points. In addition, LLMs personality profile was analyzed and compared to human normative data. The findings revealed varying levels of inter-rater agreement in the LLMs responses over a short time, with some LLMs showing higher agreement (e.g., LIama3 and GPT-4o) compared to others (e.g., GPT-4 and Gemini). Furthermore, agreement depended on used instruments as well as on domain or trait. This implies the variable robustness in LLMs' ability to reliably simulate stable personality characteristics. In the case of scales which showed at least fair agreement, LLMs displayed mostly a socially desirable profile in both agentic and communal domains, as well as a prosocial personality profile reflected in higher agreeableness and conscientiousness and lower Machiavellianism. Exhibiting temporal stability and coherent responses on personality traits is crucial for AI systems due to their societal impact and AI safety concerns.

研究动机与目标

• 促使理解LLM人格评估随时间和不同量表的稳定性程度。
• 评估对LLM对人格量表回答的评估者之间的一致性。
• 将LLM的人格特征与人类常模数据进行比较。
• 识别哪些模型和量表能产生更可靠的人格特质信号。
• 强调一致性人格模拟对AI安全与社会影响的意义。

提出的方法

• 在两个时间点对七个LLM使用标准化的人格量表进行评估。
• 衡量评估者之间的评估一致性。
• 分析一致性如何取决于量表和特质领域。
• 将LLM派生的人格特征与人类常模数据进行比较。
• 评估回答的时间稳定性及跨特质的一致性。

实验结果

研究问题

• RQ1 LLMs在两次时间点的个性测试回答是否表现出时间稳定性？
• RQ2 在评估LLM回答时，不同评估者之间的评估一致性有多大？
• RQ3 一致性是否因量表或特质领域（主动性与群体性）而异？
• RQ4 LLM的人格特征是否呈现社会可取性（社会期望），并且与人类常模相比如何？
• RQ5 哪些模型在人格评估中表现出更高的稳定性和一致性？

主要发现

• 对LLM回答的评估者间一致性在短时间内存在变化，某些模型比其他模型表现出更高的一致性。
• 一致性取决于所使用的量表以及人格领域或特质。
• 在至少达到可接受一致性的量表中，LLMs在主动性和群体性领域倾向于社会可取的特征。
• LLMs呈现亲社会人格特征，具有更高的宜人性和尽责性，较低的马基雅维利主义。
• 时间稳定性有限，这对AI安全以及持续的人格模拟的社会影响带来考量。