[论文解读] Determinants of LLM-assisted Decision-Making

Eva Eigner, T. Handler|arXiv (Cornell University)|Feb 27, 2024

Impact of AI and Big Data on Business and Society被引用 25

一句话总结

一份全面的综合性文献综述，确定了 LLM 支援的决策中的技术、心理以及决策特定因素，并提出它们相互作用的依赖框架。

ABSTRACT

Decision-making is a fundamental capability in everyday life. Large Language Models (LLMs) provide multifaceted support in enhancing human decision-making processes. However, understanding the influencing factors of LLM-assisted decision-making is crucial for enabling individuals to utilize LLM-provided advantages and minimize associated risks in order to make more informed and better decisions. This study presents the results of a comprehensive literature analysis, providing a structural overview and detailed analysis of determinants impacting decision-making with LLM support. In particular, we explore the effects of technological aspects of LLMs, including transparency and prompt engineering, psychological factors such as emotions and decision-making styles, as well as decision-specific determinants such as task difficulty and accountability. In addition, the impact of the determinants on the decision-making process is illustrated via multiple application scenarios. Drawing from our analysis, we develop a dependency framework that systematizes possible interactions in terms of reciprocal interdependencies between these determinants. Our research reveals that, due to the multifaceted interactions with various determinants, factors such as trust in or reliance on LLMs, the user's mental model, and the characteristics of information processing are identified as significant aspects influencing LLM-assisted decision-making processes. Our findings can be seen as crucial for improving decision quality in human-AI collaboration, empowering both users and organizations, and designing more effective LLM interfaces. Additionally, our work provides a foundation for future empirical investigations on the determinants of decision-making assisted by LLMs.

研究动机与目标

• 在技术、心理和任务特定维度上表征影响 LLM 支持下决策的因素。
• 整合文献以绘制因素之间的相互作用和依赖关系。
• 开发一个依赖框架，以系统化地展示因素如何相互影响 LLM 辅助决策。
• 说明因素如何影响人机协作中的决策质量和风险管理。

提出的方法

• 开展综合性文献综述，以识别影响 LLM 辅助决策的因素。
• 从技术、心理和决策特定的角度分析这些因素。
• 使用结构化符号推导一个依赖框架，以表示因素之间的相互依赖。
• 提供基于情景的示例，展示因素在实际情境中的作用。

Figure 1: Key stages in the decision-making process oriented to Simon [ 169 ] extended by LLM support options.

实验结果

研究问题

• RQ1 在技术、心理和决策特定领域，哪些因素影响 LLM 辅助决策？
• RQ2 这些因素如何相互作用并相互依赖，影响决策质量和风险管理？
• RQ3 依赖框架如何帮助设计者和组织改进人机协作中的决策？
• RQ4 哪些情景能说明所识别因素对 LLM 辅助决策的影响？

主要发现

• 对 LLM 的信任或依赖、用户的心智模型，以及信息处理特征成为 LLM 辅助决策的重要决定因素。
• 因素以互惠方式相互作用，证明了建立一个用于建模相互依赖关系的依赖框架的必要性。
• 技术性决定因素（例如，LLM 能力、透明度、提示工程）影响决策结果及风险暴露。
• 心理决定因素（如情绪、决策风格）影响用户如何参与并解读 LLM 输出。
• 决策特定因素（如任务复杂性、问责制）调节 LLM 帮助的有用性和安全性。
• 应用情景（S1–S6）展示了因素在真实世界决策情境中的发挥，并验证了该框架。

Figure 2: Process of the applied methodological approach.