[论文解读] Learning Generalizable Agents via Saliency-Guided Features Decorrelation

Sili Huang, Yanchao Sun+6 more|arXiv (Cornell University)|Oct 8, 2023

Reinforcement Learning in RoboticsComputer Science被引用 3

一句话总结

本文提出了一种显著性引导的特征去相关方法（Saliency-Guided Features Decorrelation, SGFD），这是一种样本重加权方法，利用随机傅里叶特征（Random Fourier Features, RFF）和显著性图来减少视觉强化学习中的虚假相关性，使智能体能够更好地区分任务无关与任务相关特征对决策的因果影响。SGFD在DMControl和Causal World基准测试中显著提升了对多样化视觉变化的泛化能力，优于当前最先进方法。

ABSTRACT

In visual-based Reinforcement Learning (RL), agents often struggle to generalize well to environmental variations in the state space that were not observed during training. The variations can arise in both task-irrelevant features, such as background noise, and task-relevant features, such as robot configurations, that are related to the optimal decisions. To achieve generalization in both situations, agents are required to accurately understand the impact of changed features on the decisions, i.e., establishing the true associations between changed features and decisions in the policy model. However, due to the inherent correlations among features in the state space, the associations between features and decisions become entangled, making it difficult for the policy to distinguish them. To this end, we propose Saliency-Guided Features Decorrelation (SGFD) to eliminate these correlations through sample reweighting. Concretely, SGFD consists of two core techniques: Random Fourier Functions (RFF) and the saliency map. RFF is utilized to estimate the complex non-linear correlations in high-dimensional images, while the saliency map is designed to identify the changed features. Under the guidance of the saliency map, SGFD employs sample reweighting to minimize the estimated correlations related to changed features, thereby achieving decorrelation in visual RL tasks. Our experimental results demonstrate that SGFD can generalize well on a wide range of test environments and significantly outperforms state-of-the-art methods in handling both task-irrelevant variations and task-relevant variations.

研究动机与目标

• 解决在面对任务无关与任务相关特征的未见环境变化时，视觉强化学习中泛化能力差的挑战。
• 解耦高维视觉观测中固有特征相关性所引发的输入特征与决策之间的虚假相关性。
• 使智能体能够准确关联发生变化的特征（如背景噪声或机器人构型）与其对最优决策的真实影响。
• 通过基于显著性图和非线性相关性估计的样本重加权，而非仅依赖不变表示学习，来提升策略的泛化能力。

提出的方法

• SGFD使用随机傅里叶特征（RFF）以线性计算复杂度近似高维视觉特征之间的复杂非线性相关性。
• 训练一个分类模型以识别输入图像的来源（例如环境或变化类型），其显著性图突出显示了在不同环境中发生变化的特征。
• 显著性图指导样本重加权，以专门最小化变化特征的估计相关性，从而降低其对策略学习的混淆影响。
• 利用RFF嵌入特征与动作之间的互相关矩阵来估计非线性特征-决策相关性，随后通过重加权最小化这些相关性。
• 在训练过程中应用样本重加权，以减少相关但非因果特征的影响，同时保留与决策相关变化的关联。
• 该方法在DMControl和Causal World基准上进行评估，并通过消融实验验证了RFF和显著性引导的作用。

实验结果

研究问题

• RQ1 基于显著性图引导与RFF相关性估计的样本重加权，是否能提升在任务无关与任务相关环境变化下的视觉强化学习泛化能力？
• RQ2 RFF在捕捉高维视觉特征之间非线性相关性以实现去相关方面有多高效？
• RQ3 显著性引导的重加权在多大程度上提升了模型区分因果与虚假特征-决策关联的能力？
• RQ4 仅对显著性图识别为变化的特征进行去相关，是否比对全部特征进行去相关能带来更好的泛化效果？
• RQ5 SGFD在处理两类环境变化方面，与现有不变表示学习方法相比表现如何？

主要发现

• SGFD在DMControl基准中，对包含干扰背景视频和不同机器人参数的环境，显著优于当前最先进方法的泛化性能。
• 消融实验表明，移除RFF会导致性能显著下降，证实其在建模复杂非线性相关性中的关键作用。
• 移除显著性引导模型后性能急剧下降，表明其在聚焦于相关变化特征的去相关方面至关重要。
• 重加权过程成功平衡了不同环境中相似状态实例（如‘跌倒’姿势）的数量，消除了虚假相关性。
• 样本权重的可视化结果证实，SGFD有效降低了任务无关特征（如背景）与策略决策之间的相关性。
• 该方法在包括背景噪声和机器人构型变化在内的多样化视觉变化中均表现出良好的泛化能力，且在DMControl和Causal World基准上均得到验证。