[论文解读] Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation

Xingning Dong, Tian Gan+4 more|arXiv (Cornell University)|Mar 18, 2022

Multimodal Machine Learning Applications被引用 6

一句话总结

该论文提出了一种新颖的堆叠混合注意力（Stacked Hybrid-Attention, SHA）编码器和群体协同学习（Group Collaborative Learning, GCL）解码器，以解决场景图生成中模态融合不足和关系预测偏差的问题。通过堆叠自注意力与交叉注意力模块以增强视觉-语义交互，并采用基于分组增量式分类器设置的并行优化与知识蒸馏机制，该方法在Visual Genome（VG）和GQA数据集上取得了新的最先进性能，几乎使无偏置指标下的性能较基线方法翻倍。

ABSTRACT

Scene Graph Generation, which generally follows a regular encoder-decoder pipeline, aims to first encode the visual contents within the given image and then parse them into a compact summary graph. Existing SGG approaches generally not only neglect the insufficient modality fusion between vision and language, but also fail to provide informative predicates due to the biased relationship predictions, leading SGG far from practical. Towards this end, in this paper, we first present a novel Stacked Hybrid-Attention network, which facilitates the intra-modal refinement as well as the inter-modal interaction, to serve as the encoder. We then devise an innovative Group Collaborative Learning strategy to optimize the decoder. Particularly, based upon the observation that the recognition capability of one classifier is limited towards an extremely unbalanced dataset, we first deploy a group of classifiers that are expert in distinguishing different subsets of classes, and then cooperatively optimize them from two aspects to promote the unbiased SGG. Experiments conducted on VG and GQA datasets demonstrate that, we not only establish a new state-of-the-art in the unbiased metric, but also nearly double the performance compared with two baselines.

研究动机与目标

• 解决现有场景图生成（SGG）模型中视觉与语言模态间融合有限的问题，该问题阻碍了准确的关系预测。
• 克服由谓词分布长尾导致的关系预测偏差，即头部谓词占主导而尾部谓词被严重低估的问题。
• 开发一种模型无关的去偏策略，以提升所有谓词类别（尤其是罕见类别）的泛化能力。
• 通过增强编码器表征能力和解码器决策机制，为下游视觉-语言任务生成更具信息量且更均衡的场景图。

提出的方法

• 提出一种堆叠混合注意力（SHA）网络，整合自注意力（SA）用于模态内特征优化，以及交叉注意力（CA）用于模态间交互，并在多层中堆叠以增强特征表示能力。
• 设计一种群体协同学习（GCL）策略，根据训练实例频率将高度不平衡的谓词集合划分为相对平衡的组别。
• 实现一种分组增量式分类空间，其中每个分类器在谓词子集上进行训练，并逐步扩展先前的分类空间。
• 引入中位数重采样（Median Re-Sampling）以平衡每个分类器的训练数据，从而提升对罕见谓词的泛化能力。
• 应用并行分类器优化（Parallel Classifier Optimization, PCO），通过共享梯度聚合联合优化所有分类器，作为弱约束以稳定训练过程。
• 开发协同知识蒸馏（Collaborative Knowledge Distillation, CKD），将早期分类器的知识迁移至后续分类器，施加强约束以保留判别能力并减少头部类别过拟合。

Figure 1 : Two intentions to promote the unbiased SGG. (1) For the insufficient modality fusion, we aim to enhance both the intra-modal refinement and the inter-modal interaction (see the top-right corner of the figure). And (2) we split the extremely unbalanced dataset into a set of relatively bala

实验结果

研究问题

• RQ1 堆叠混合注意力机制是否能有效提升场景图生成中视觉与语言特征之间的模态融合？
• RQ2 通过频率分组并为平衡子集训练专用分类器，能否缓解由长尾数据引起的谓词预测偏差？
• RQ3 在多个分类器之间实施协同优化与知识蒸馏，能否同时提升头部与尾部谓词的性能？
• RQ4 所提出的GCL策略在无偏评估指标下，相较于标准与去偏基线方法，其性能提升程度如何？

主要发现

• 所提出的SHA+GCL模型在VG150数据集上达到新的最先进性能，PredCls指标下mR@50/100分别为41.6/44.0，显著优于此前方法。
• 消融实验表明，若移除GCL，性能将下降至18.8/20.5 mR@50/100，表明GCL使性能近乎翻倍。
• 移除CKD后，头部类别性能显著下降，而CKD有助于维持较强的尾部类别预测表现，证明其在知识迁移中的有效性。
• 在SHA中移除任一CA或SA模块均导致性能显著下降，证实模态内与模态间注意力对准确预测均至关重要。
• 最优配置采用μ=4及自顶向下的知识匹配策略，在所有指标上均取得最佳整体性能。

Figure 2 : The framework of the common pipeline in SGG, which includes five key components. Notably, we improve three key components marked in red in the figure. Specifically, we propose the Stacked Hybrid-Attention network to enhance the object encoder and the relation encoder, and we also devise t