[Paper Review] Is ChatGPT a Good Personality Recognizer? A Preliminary Study

Yu Ji, Wen Wu+4 more|arXiv (Cornell University)|Jul 8, 2023

Machine Learning in Healthcare20 citations

TL;DR

The paper evaluates ChatGPT’s ability to recognize Big-Five personality traits from text using various prompting strategies, compares against baselines and SOTA models, analyzes fairness, and examines impact on downstream tasks.

ABSTRACT

In recent years, personality has been regarded as a valuable personal factor being incorporated into numerous tasks such as sentiment analysis and product recommendation. This has led to widespread attention to text-based personality recognition task, which aims to identify an individual's personality based on given text. Considering that ChatGPT has recently exhibited remarkable abilities on various natural language processing tasks, we provide a preliminary evaluation of ChatGPT on text-based personality recognition task for generating effective personality data. Concretely, we employ a variety of prompting strategies to explore ChatGPT's ability in recognizing personality from given text, especially the level-oriented prompting strategy we designed for guiding ChatGPT in analyzing given text at a specified level. The experimental results on two representative real-world datasets reveal that ChatGPT with zero-shot chain-of-thought prompting exhibits impressive personality recognition ability and is capable to provide natural language explanations through text-based logical reasoning. Furthermore, by employing the level-oriented prompting strategy to optimize zero-shot chain-of-thought prompting, the performance gap between ChatGPT and corresponding state-of-the-art model has been narrowed even more. However, we observe that ChatGPT shows unfairness towards certain sensitive demographic attributes such as gender and age. Additionally, we discover that eliciting the personality recognition ability of ChatGPT helps improve its performance on personality-related downstream tasks such as sentiment classification and stress prediction.

Motivation & Objective

• Motivate text-based personality recognition as a valuable downstream signal for NLP tasks.
• Assess ChatGPT’s ability to infer Big-Five traits from user-generated text under different prompting strategies.
• Compare ChatGPT against RNN, RoBERTa, and SOTA models on two datasets.
• Explore the interpretability of ChatGPT outputs via natural language explanations.
• Investigate fairness biases across gender and age groups and potential downstream task benefits.

Proposed method

• Use zero-shot, zero-shot chain-of-thought (CoT), and one-shot prompting to elicit personality levels (O, C, E, A, N) from text.
• Design zero-shot level-oriented CoT prompting to analyze text at word, sentence, or document level.
• Evaluate on Essays and PAN datasets against baselines (RNN, RoBERTa) and SOTA (HPMN BERT).
• Measure accuracy and compute AIP (accuracy improvement percentage) relative to SOTA.
• Provide ChatGPT outputs with natural language explanations for CoT prompts.
• Analyze fairness by incorporating gender and age prompts and visualize distribution of predicted trait levels.

Experimental results

Research questions

• RQ1 RQ1: How do different prompting strategies affect ChatGPT’s ability to identify personality from text?
• RQ2 RQ2: How unfair is ChatGPT when serving as a personality recognizer on sensitive demographic attributes?
• RQ3 RQ3: Does the inferred personality improve performance on downstream tasks such as sentiment classification and stress prediction?

Key findings

• Zero-shot prompting with CoT yields the best average performance among prompting strategies, though still below SOTA.
• Zero-shot CoT prompting enables natural language explanations and improved interpretability of predictions.
• Level-oriented CoT prompting (word/sentence/document level) can further enhance accuracy for targeted text analysis.
• ChatGPT shows demographic unfairness, with women predicted more often as high on certain traits and older individuals more often as low on Openness.
• Elicited personality traits can improve performance on downstream tasks like sentiment classification and stress prediction.
• On PAN, ChatGPT CoT_S (sentence level) and CoT_D (document level) offer notable gains in certain traits.