[论文解读] What does ChatGPT return about human values? Exploring value bias in ChatGPT using a descriptive value theory

Ronald Fischer, Markus Luczak–Roesch+1 more|arXiv (Cornell University)|Apr 7, 2023

Artificial Intelligence in Healthcare and Education被引用 10

一句话总结

本文使用施瓦茨的基本价值理论测试ChatGPT可能的价值偏见，发现对显性价值偏见很少，ChatGPT输出中动机性内容的高保真再现，以及对社会取向价值的某些合并。

ABSTRACT

There has been concern about ideological basis and possible discrimination in text generated by Large Language Models (LLMs). We test possible value biases in ChatGPT using a psychological value theory. We designed a simple experiment in which we used a number of different probes derived from the Schwartz basic value theory (items from the revised Portrait Value Questionnaire, the value type definitions, value names). We prompted ChatGPT via the OpenAI API repeatedly to generate text and then analyzed the generated corpus for value content with a theory-driven value dictionary using a bag of words approach. Overall, we found little evidence of explicit value bias. The results showed sufficient construct and discriminant validity for the generated text in line with the theoretical predictions of the psychological model, which suggests that the value content was carried through into the outputs with high fidelity. We saw some merging of socially oriented values, which may suggest that these values are less clearly differentiated at a linguistic level or alternatively, this mixing may reflect underlying universal human motivations. We outline some possible applications of our findings for both applications of ChatGPT for corporate usage and policy making as well as future research avenues. We also highlight possible implications of this relatively high-fidelity replication of motivational content using a linguistic model for the theorizing about human values.

研究动机与目标

• 评估ChatGPT输出是否呈现符合心理价值理论（施瓦茨基本价值）的价值偏见。
• 评估ChatGPT生成的文本是否保留由描述性价值理论定义的动机内容。
• 探索在企业使用、政策制定以及语言模型中人类价值观未来研究的影响。

提出的方法

• 设计一个探测性实验，使用源自施瓦茨基本价值理论的提示（修订版个人价值问卷项、价值类型定义、价值名称）。
• 通过OpenAI API多次向ChatGPT发出提示以生成文本。
• 使用基于理论的价值词典结合词袋模型对生成的语料进行分析。

实验结果

研究问题

• RQ1 ChatGPT输出是否如施瓦茨理论所定义，表现出显性价值偏见？
• RQ2 输出中的价值内容在多大程度上忠实再现了提示中编码的动机性内容？
• RQ3 是否存在会影响对生成文本中价值区分的语言模式（例如社会取向价值的合并）？
• RQ4 对企业使用、政策制定以及未来关于价值相关语言模型的研究有何影响？

主要发现

• 总体而言，ChatGPT输出中显性价值偏见的证据很少。
• 生成的文本展示了与心理模型相一致的充分构念效度和判别效度。
• 价值内容似乎以高保真度被传递到输出中。
• 观察到对社会取向价值的某些合并，这可能反映语言上的模糊性或普遍的人类动机。