[论文解读] Hierarchical Local-Global Transformer for Temporal Sentence Grounding

Xiang Fang, Daizong Liu+3 more|arXiv (Cornell University)|Aug 31, 2022

Multimodal Machine Learning Applications被引用 5

一句话总结

该论文提出了一种分层局部-全局Transformer（HLGT）用于时序句子定位，通过双分支Transformer编码器和全局-局部交互模块，建模视频片段与句子短语之间的多粒度交互，实现了在三个基准数据集上的最先进性能，准确率和计算效率均得到提升。

ABSTRACT

This paper studies the multimedia problem of temporal sentence grounding (TSG), which aims to accurately determine the specific video segment in an untrimmed video according to a given sentence query. Traditional TSG methods mainly follow the top-down or bottom-up framework and are not end-to-end. They severely rely on time-consuming post-processing to refine the grounding results. Recently, some transformer-based approaches are proposed to efficiently and effectively model the fine-grained semantic alignment between video and query. Although these methods achieve significant performance to some extent, they equally take frames of the video and words of the query as transformer input for correlating, failing to capture their different levels of granularity with distinct semantics. To address this issue, in this paper, we propose a novel Hierarchical Local-Global Transformer (HLGT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities for learning more fine-grained multi-modal representations. Specifically, we first split the video and query into individual clips and phrases to learn their local context (adjacent dependency) and global correlation (long-range dependency) via a temporal transformer. Then, a global-local transformer is introduced to learn the interactions between the local-level and global-level semantics for better multi-modal reasoning. Besides, we develop a new cross-modal cycle-consistency loss to enforce interaction between two modalities and encourage the semantic alignment between them. Finally, we design a brand-new cross-modal parallel transformer decoder to integrate the encoded visual and textual features for final grounding. Extensive experiments on three challenging datasets show that our proposed HLGT achieves a new state-of-the-art performance.

研究动机与目标

• 解决现有基于Transformer的方法在同时捕捉局部（片段/短语）和全局（视频/查询）语义方面的局限性。
• 克服标准Transformer中帧与词等权重交互导致的部分语义匹配问题。
• 实现无需耗时后处理或提议生成的端到端时序句子定位。
• 通过新颖的跨模态循环一致性损失和并行解码器，提升多模态对齐。
• 通过一致的模型性能和效率，增强在多样化视频领域中的泛化能力。

提出的方法

• 将视频和句子分别划分为片段和短语，通过时序Transformer学习局部依赖关系。
• 引入全局-局部Transformer，建模局部级别与全局级别表示之间的交互。
• 设计一种跨模态循环一致性损失（$L_{CMCC}$），强制视觉与文本模态之间的语义对齐。
• 采用并行跨模态Transformer解码器，整合视觉与文本特征以进行最终段落预测。
• 使用可学习的集合预测头结合置信度分数，实现无需后处理的端到端定位。
• 优化超参数（$\lambda_{\ell_1}$, $\lambda_{{IoU}}$, $\lambda_f$），以平衡定位、置信度和IoU监督。

实验结果

研究问题

• RQ1 建模多粒度交互（片段/短语与视频/查询）是否能提升时序句子定位的准确性？
• RQ2 分层局部-全局注意力机制是否优于将所有帧与词同等对待的标准Transformer？
• RQ3 循环一致的跨模态损失是否能增强视频与文本之间的语义对齐？
• RQ4 所提方法在多样化视频领域（如烹饪、室内、开放世界）中的泛化能力如何？
• RQ5 该模型能否在端到端训练和最小后处理下实现最先进性能？

主要发现

• HLGT在ActivityNet Captions、Charades-STA和TACoS数据集上均取得新的最先进性能，R@5、IoU=0.5的得分分别为67.92%、84.19%和94.50%。
• 所提出的CMCC损失在TACoS上相比余弦相似度损失提升1.68%（R@5、IoU=0.5）。
• 最优超参数为$\lambda_{\ell_1}=0.8$、$\lambda_{{IoU}}=0.5$和$\lambda_f=0.2$，实现定位与置信度的平衡。
• HLGT每秒可处理190.76至257.38个视频，展现出在各数据集上的高计算效率。
• 定性结果表明，HLGT通过捕捉多粒度视觉与语言上下文，相比VIDGTR实现了更精确的定位。
• 该模型在参数量相近（114–119M）的情况下，跨数据集保持一致的性能表现，表明其具有强大的泛化能力。