[论文解读] Collaborative Group: Composed Image Retrieval via Consensus Learning from Noisy Annotations

Xu Zhang, Zhedong Zheng+2 more|arXiv (Cornell University)|Jun 3, 2023

Multimodal Machine Learning Applications被引用 5

一句话总结

本文提出了一种名为共识网络（Css-Net）的新框架，用于语言引导的图像检索，通过训练四个不同的组合器从噪声三元组中学习，并利用KL散度损失达成共识，从而缓解由噪声和不完整标注引起的三元组歧义问题。Css-Net实现了最先进性能，在FashionIQ数据集上将R@10提升+2.77%，R@50提升+6.67%。

ABSTRACT

Composed image retrieval extends content-based image retrieval systems by enabling users to search using reference images and captions that describe their intention. Despite great progress in developing image-text compositors to extract discriminative visual-linguistic features, we identify a hitherto overlooked issue, triplet ambiguity, which impedes robust feature extraction. Triplet ambiguity refers to a type of semantic ambiguity that arises between the reference image, the relative caption, and the target image. It is mainly due to the limited representation of the annotated text, resulting in many noisy triplets where multiple visually dissimilar candidate images can be matched to an identical reference pair (i.e., a reference image + a relative caption). To address this challenge, we propose the Consensus Network (Css-Net), inspired by the psychological concept that groups outperform individuals. Css-Net comprises two core components: (1) a consensus module with four diverse compositors, each generating distinct image-text embeddings, fostering complementary feature extraction and mitigating dependence on any single, potentially biased compositor; (2) a Kullback-Leibler divergence loss that encourages learning of inter-compositor interactions to promote consensual outputs. During evaluation, the decisions of the four compositors are combined through a weighting scheme, enhancing overall agreement. On benchmark datasets, particularly FashionIQ, Css-Net demonstrates marked improvements. Notably, it achieves significant recall gains, with a 2.77% increase in R@10 and 6.67% boost in R@50, underscoring its competitiveness in addressing the fundamental limitations of existing methods.

研究动机与目标

• 为解决语言引导图像检索中的三元组歧义问题，即标注者遗漏图像的细粒度细节，导致同一查询存在多个错误负样本。
• 通过允许多个组合器协同学习并在训练和推理过程中达成共识，减少由噪声三元组引起的模型偏差。
• 通过最小化模糊且不完整的标注的负面影响，改进基于度量学习的表示学习。
• 开发一种鲁棒且可泛化的技术，无需对现有模型进行架构上的重大修改即可提升性能。

提出的方法

• Css-Net采用包含四个不同组合器的共识模块，每个组合器使用不同的融合策略生成多样化的图文嵌入。
• 每个组合器均采用金字塔式训练范式和辅助的图文组合器设计，以确保其获得独特的知识。
• 应用Kullback-Leibler（KL）散度损失，以促进知识蒸馏并减少组合器之间的个体偏差。
• 在训练过程中，组合器通过最小化彼此预测之间的KL散度来相互学习，从而在噪声数据上达成共识。
• 在推理阶段，对所有四个组合器的决策进行加权聚合，以提升鲁棒性并减少个体错误。
• 该方法与现有架构正交，可与强基线模型（如CoSMo）集成，进一步提升性能。

Figure 1. Illustration of the language-guided image retrieval system. Using a reference image and an associated descriptive sentence, the system endeavors to accurately retrieve the intended target image from candidate images for user convenience.

实验结果

研究问题

• RQ1 由不完整标注引起的三元组歧义在多大程度上影响基于度量学习的语言引导图像检索性能？
• RQ2 在噪声三元组上训练的多个组合器能否通过共识学习共同降低偏差并提升鲁棒性？
• RQ3 基于共识的框架在处理模糊且噪声的图文三元组时，其性能相对于单个模型能提升多少？
• RQ4 所提出的基于KL的蒸馏机制是否能有效减少由噪声标注引起的模型偏差？
• RQ5 该共识方法是否能在不同数据集上泛化，并在检索基准测试中超越最先进方法？

主要发现

• 与CLIP4cir相比，Css-Net在FashionIQ基准上将R@10提升+2.77%，R@50提升+6.67%，显著改善了对三元组歧义的处理能力。
• 在Shoes数据集上，Css-Net相比最先进方法CLVC-Net实现R@1提升+2.49%，R@10提升+2.42%，证实了其在不同数据集上的鲁棒性。
• 在FashionIQ的Dress子集上，Css-Net将基线模型的R@10提升+4.68%，表明即使在强基线模型下也具有显著有效性。
• 联合使用全部四个组合器的推理性能优于任一单个组合器，验证了共识策略的有效性。
• 在Fashion200k数据集上，Css-Net在模型集成下实现23.4%的R@1和52.0%的R@10，尽管基线存在局限，仍优于许多SOTA方法。
• 消融实验表明，KL损失和多组合器设计对于减少偏差并在噪声数据上提升泛化能力至关重要。

Figure 2. Illustration of the triplet ambiguity problem. Triplet ambiguity denotes multiple false-negative samples in the dataset. It is due to the annotator usually seeing one triplet with true match (​ ) at a time, while neglecting other candidates ( ) in the whole dataset. Triplet ambiguity large