[论文解读] Self-Emotion-Mediated Exploration in Artificial Intelligence Mirrors: Findings from Cognitive Psychology

Gustavo Assunção, Miguel Castelo‐Branco+1 more|arXiv (Cornell University)|Feb 13, 2023

Child and Animal Learning Development被引用 5

一句话总结

本文提出了一种新颖的人工智能框架，通过建模认知心理学中的认识性情绪（惊讶）和成就情绪（自豪）来实现自我情绪驱动的探索。该框架利用准确率和置信度等性能指标来表征这些情绪，并通过可微分函数在深度强化学习架构中模拟这些情绪。由此，智能体能够自主学习探索行为，展现出内在情绪状态与探索行为之间的因果关系，其表现与人类认知模式高度一致，90%的智能体成功习得探索策略。

ABSTRACT

Background: Exploration of the physical environment is an indispensable precursor to information acquisition and knowledge consolidation for living organisms. Yet, current artificial intelligence models lack these autonomy capabilities during training, hindering their adaptability. This work proposes a learning framework for artificial agents to obtain an intrinsic exploratory drive, based on epistemic and achievement emotions triggered during data observation. Methods: This study proposes a dual-module reinforcement framework, where data analysis scores dictate pride or surprise, in accordance with psychological studies on humans. A correlation between these states and exploration is then optimized for agents to meet their learning goals. Results: Causal relationships between states and exploration are demonstrated by the majority of agents. A 15.4\% mean increase is noted for surprise, with a 2.8\% mean decrease for pride. Resulting correlations of $ρ_{surprise}=0.461$ and $ρ_{pride}=-0.237$ are obtained, mirroring previously reported human behavior. Conclusions: These findings lead to the conclusion that bio-inspiration for AI development can be of great use. This can incur benefits typically found in living beings, such as autonomy. Further, it empirically shows how AI methodologies can corroborate human behavioral findings, showcasing major interdisciplinary importance. Ramifications are discussed.

研究动机与目标

• 通过将类人情绪驱动力嵌入人工智能智能体，弥合人工智能与生物探索之间的差距。
• 基于认知心理学原理，利用可测量的性能指标建模认识性情绪与成就情绪——惊讶与自豪。
• 通过将内在情绪状态与探索行为相联系，使人工智能体能够自主探索环境，从而提升适应性与学习效率。
• 验证人工智能中的情绪驱动探索是否能复现认知心理学实验中观察到的人类行为模式。

提出的方法

• 情绪被建模为可微分函数：自豪作为任务准确率的非线性函数，惊讶作为准确率与置信度得分的马鞍形函数。
• 该框架采用深度强化学习架构，包含演员-评论家网络与回放缓冲区，以稳定训练并支持时序差分学习。
• 通过截断处理将情绪函数限制在[0,1]区间，并引入高斯噪声以模拟个体差异，确保情绪表现的现实多样性。
• 评论家网络使用目标网络（软更新率τ=0.005）计算时序差分目标，损失函数通过Adam优化器以0.002的学习率进行最小化。
• 惊讶通过准确率与置信度差值平方的旋转与平移函数建模，有效捕捉高置信度错误与低置信度成功作为高惊讶状态。
• 自豪被建模为受高斯影响的准确率函数，峰值出现在高准确率处，反映基于成就的情绪反应。

Figure 1 : Curves demonstrating how the emotion of pride may correlate with accuracy. Example curves following a positive prediction of pride based on increasing accuracy, as described by cognitive psychology research [ 32 , 31 ] . Considering how increasing task accuracy equates to personal achieve

实验结果

研究问题

• RQ1 人工智能智能体能否基于源自认知心理学的自我生成情绪状态来调节探索行为？
• RQ2 人工智能中的情绪驱动探索策略在多大程度上复现了认知心理学研究中观察到的人类行为模式？
• RQ3 准确率与置信度等性能指标如何映射为惊讶与自豪等人工情绪，从而有效驱动探索？
• RQ4 当由显式建模的内在情绪状态引导时，深度强化学习智能体能否实现探索行为的收敛？

主要发现

• 90%的人工智能智能体成功学习到基于内在情绪状态调节探索行为，证明了情绪与行为之间的因果关系。
• 惊讶函数有效捕捉了高置信度错误与低置信度成功作为高惊讶状态，与认知心理学研究结果一致。
• 自豪函数与准确率呈现非线性正相关关系，反映出与人类研究中观察到的基于成就的情绪反应一致。
• 具备个性化情绪参数（通过随机C1与C2值设定）的智能体表现出多样化但一致的探索行为，模拟了人格差异。
• 采用截断并注入噪声的情绪函数确保了所有智能体的情绪输出稳定且有界，提升了训练鲁棒性。
• 将情绪函数整合进基于DDPG的强化学习框架后，100个训练周期内探索行为均实现一致收敛。

Figure 2 : Multiple perspective surface view demonstrating how the emotion of surprise correlates with accuracy and confidence. A potential surface representative of how surprise fluctuates with polarized variations of task accuracy and agent confidence, as described by cognitive psychology research