[论文解读] Improving Image Captioning Descriptiveness by Ranking and LLM-based Fusion

Luigi Celona, Simone Bianco+2 more|arXiv (Cornell University)|Jun 20, 2023

Multimodal Machine Learning Applications被引用 4

一句话总结

本文提出了一种无需训练的方法，通过使用图像-文本匹配度量对五种最先进（SoTA）图像字幕模型的输出进行排序，再利用大语言模型（LLM）融合排名靠前的两个字幕，从而生成更具描述性的图像字幕。该方法生成的字幕在语义丰富度、多样性以及人类偏好方面均优于单一的SoTA模型，在MS-COCO数据集上展现出与人类判断更高的对齐度。

ABSTRACT

State-of-The-Art (SoTA) image captioning models are often trained on the MicroSoft Common Objects in Context (MS-COCO) dataset, which contains human-annotated captions with an average length of approximately ten tokens. Although effective for general scene understanding, these short captions often fail to capture complex scenes and convey detailed information. Moreover, captioning models tend to exhibit bias towards the ``average'' caption, which captures only the more general aspects, thus overlooking finer details. In this paper, we present a novel approach to generate richer and more informative image captions by combining the captions generated from different SoTA captioning models. Our proposed method requires no additional model training: given an image, it leverages pre-trained models from the literature to generate the initial captions, and then ranks them using a newly introduced image-text-based metric, which we name BLIPScore. Subsequently, the top two captions are fused using a Large Language Model (LLM) to produce the final, more detailed description. Experimental results on the MS-COCO and Flickr30k test sets demonstrate the effectiveness of our approach in terms of caption-image alignment and hallucination reduction according to the ALOHa, CAPTURE, and Polos metrics. A subjective study lends additional support to these results, suggesting that the captions produced by our model are generally perceived as more consistent with human judgment. By combining the strengths of diverse SoTA models, our method enhances the quality and appeal of image captions, bridging the gap between automated systems and the rich and informative nature of human-generated descriptions. This advance enables the generation of more suitable captions for the training of both vision-language and captioning models.

研究动机与目标

• 为解决当前图像字幕模型在生成丰富、详细描述时的局限性，该局限性源于其依赖于MS-COCO数据集中短篇幅、平均长度的人工标注字幕。
• 探究融合多个SoTA模型输出是否能生成比单个模型更具描述性和代表性的字幕。
• 评估生成的字幕是否与人类判断更一致，以及是否更适合作为视觉-语言模型的高质量训练数据。
• 探究当前图像字幕模型是否可通过在更具描述性、多样性的真实标注数据上进行训练而得到改进。

提出的方法

• 选取五种高性能且可复现的SoTA图像字幕模型：BLIP-2、ExpNet-v2、GIT、OFA和ViT-GPT2。
• 使用所选模型为每张图像生成多个字幕。
• 利用图像-文本匹配（ITM）度量对生成的字幕进行排序，以评估其与输入图像的对齐程度。
• 为每张图像选择排名前两位的字幕作为LLM融合步骤的输入。
• 使用大语言模型（如GPT-3.5 Davinci）将前两名字幕的语义融合为一个更富描述性的单一字幕。
• 采用经过精心设计的提示词，以激发上下文感知、自然流畅的融合效果，同时避免重复的结构模式。

Figure 1: Overview of the proposed image captioning pipeline.

实验结果

研究问题

• RQ1 RQ1：融合多个SoTA模型的字幕是否能生成比单个模型更详细、语义更丰富的字幕？
• RQ2 RQ2：根据人类判断，融合后的字幕是否更准确地反映了图像内容？
• RQ3 RQ3：若当前图像字幕模型在本方法生成的高质量、描述性数据上进行训练，是否能生成更丰富的字幕？

主要发现

• 所提方法生成的字幕在语义丰富度和描述细节方面显著优于单一SoTA模型，该结论得到人工评估的证实。
• 人工评估者更偏好融合后的字幕，而非任一单一SoTA模型的输出，表明其感知质量与真实感更高。
• 该模型在MS-COCO测试集上与人类判断的一致性更高，展现出与人类感知更佳的对齐效果。
• 基于LLM的融合过程引入了原始输入字幕中不存在的情感与上下文深度，例如“一个引人入胜的场景”或“生动而充满活力”的表达。
• 尽管效果显著，该方法在部分情况下仍存在模式崩溃问题，当多个模型生成完全相同或近似相同的字幕时，多样性提升受限。
• 在16张测试图像中，LLM引入了结构化的前缀（“这张图像的字幕可能是：”），该前缀并非输入字幕的一部分，而是提示工程的结果。

Figure 3: Image-text matching scores for the captions generated by using the five considered models on an MS-COCO test image. The captioning models are reported in descending order with respect to the BLIPScore.