[论文解读] Locate then Segment: A Strong Pipeline for Referring Image Segmentation

Ya Jing, Tao Kong+4 more|arXiv (Cornell University)|Mar 30, 2021

Multimodal Machine Learning Applications参考文献 40被引用 11

一句话总结

该论文提出了一种简单而有效的‘定位然后分割’（LTS）流水线用于指代图像分割，将任务解耦为显式的对象定位，随后进行高精度分割。通过融合视觉与语言特征，利用定位模块预先预测对象位置，并应用轻量级分割头，LTS在RefCOCO+上实现SOTA性能，提升+3.2%，在RefCOCOg上提升+3.4%。

ABSTRACT

Referring image segmentation aims to segment the objects referred by a natural language expression. Previous methods usually focus on designing an implicit and recurrent feature interaction mechanism to fuse the visual-linguistic features to directly generate the final segmentation mask without explicitly modeling the localization information of the referent instances. To tackle these problems, we view this task from another perspective by decoupling it into a "Locate-Then-Segment" (LTS) scheme. Given a language expression, people generally first perform attention to the corresponding target image regions, then generate a fine segmentation mask about the object based on its context. The LTS first extracts and fuses both visual and textual features to get a cross-modal representation, then applies a cross-model interaction on the visual-textual features to locate the referred object with position prior, and finally generates the segmentation result with a light-weight segmentation network. Our LTS is simple but surprisingly effective. On three popular benchmark datasets, the LTS outperforms all the previous state-of-the-art methods by a large margin (e.g., +3.2% on RefCOCO+ and +3.4% on RefCOCOg). In addition, our model is more interpretable with explicitly locating the object, which is also proved by visualization experiments. We believe this framework is promising to serve as a strong baseline for referring image segmentation.

研究动机与目标

• 为解决指代图像分割中隐式、循环特征融合的局限性，该方法通常无法显式建模对象定位。
• 通过跨模态交互显式预测被指对象的空间位置，以提升分割精度。
• 开发一种简单、可解释且高效的流水线，超越复杂SOTA模型。
• 通过将优化直接集成到网络中，减少对后处理技术（如DCRF）的依赖。

提出的方法

• 该方法首先使用早期融合将视觉与语言特征融合为跨模态表示。
• 定位模块通过相关性过滤，基于语言表达识别最相关的图像区域，生成位置先验。
• 将位置先验与跨模态特征拼接，以引导分割网络。
• 使用轻量级全卷积网络（FCN）生成最终的高分辨率分割掩码。
• 通过结合分割与定位监督的多任务损失进行训练，以提升对齐效果。
• 该框架采用简单主干网络（如ResNet），并在主干路径中避免使用复杂的循环或注意力机制。

Figure 1: The illustration of referring image segmentation. Given a referring expression and an image, the model aims to generate a segmentation mask of the corresponding object in image referred by the language expression. Best viewed in color.

实验结果

研究问题

• RQ1 显式对象定位是否能提升指代图像分割中的分割精度？
• RQ2 与端到端融合方法相比，解耦定位与分割是否能带来更好的性能与可解释性？
• RQ3 基于语言引导特征过滤生成的位置先验如何影响掩码质量？
• RQ4 轻量级、非循环架构能否在此任务中超越复杂循环模型？

主要发现

• LTS在RefCOCO上达到66.75% IoU，在RefCOCO+上达到54.94%，在RefCOCOg上达到54.51%，创下新的SOTA性能。
• 与先前SOTA结果相比，LTS在RefCOCO+上提升+3.2%，在RefCOCOg上提升+3.4%。
• 消融实验确认，定位模块与分割头均对性能提升有显著贡献。
• 使用更大的输入分辨率（480×480）可进一步提升性能，在RefCOCO验证集上达到65.90% IoU。
• 将过滤模块替换为基于Transformer的定位模块可进一步提升性能，表明更强注意力机制的优势。
• 该方法更具可解释性，可视化显示模型在分割前明确关注正确对象区域。

Figure 2: The architecture of our proposed method. The visual feature and linguistic feature are extracted by a deep convolutional network (ConvNet) and a bi-GRU network respectively, and then fused to generate the cross-modal features. Next a cross-modal interaction module (e.g., filtering and tran