[论文解读] Match-Ignition: Plugging PageRank into Transformer for Long-form Text Matching

Liang Pang, Yanyan Lan+1 more|arXiv (Cornell University)|Jan 16, 2021

Topic Modeling参考文献 33被引用 9

一句话总结

Match-Ignition 提出了一种分层噪声过滤框架，将 PageRank 集成到 Transformer 架构中，通过过滤无关的句子和词语来提升长文本匹配性能。通过基于相似度得分和注意力得分分别构建句子图与词语图，并应用 PageRank 识别关键内容，该模型在长文本匹配任务中实现了最先进性能，同时提升了效率。

ABSTRACT

Neural text matching models have been widely used in community question answering, information retrieval, and dialogue. However, these models designed for short texts cannot well address the long-form text matching problem, because there are many contexts in long-form texts can not be directly aligned with each other, and it is difficult for existing models to capture the key matching signals from such noisy data. Besides, these models are computationally expensive for simply use all textual data indiscriminately. To tackle the effectiveness and efficiency problem, we propose a novel hierarchical noise filtering model, namely Match-Ignition. The main idea is to plug the well-known PageRank algorithm into the Transformer, to identify and filter both sentence and word level noisy information in the matching process. Noisy sentences are usually easy to detect because previous work has shown that their similarity can be explicitly evaluated by the word overlapping, so we directly use PageRank to filter such information based on a sentence similarity graph. Unlike sentences, words rely on their contexts to express concrete meanings, so we propose to jointly learn the filtering and matching process, to well capture the critical word-level matching signals. Specifically, a word graph is first built based on the attention scores in each self-attention block of Transformer, and key words are then selected by applying PageRank on this graph. In this way, noisy words will be filtered out layer by layer in the matching process. Experimental results show that Match-Ignition outperforms both SOTA short text matching models and recent long-form text matching models. We also conduct detailed analysis to show that Match-Ignition efficiently captures important sentences and words, to facilitate the long-form text matching process.

研究动机与目标

• 解决语义匹配任务中长文本的噪声问题，现有为短文本设计的模型因对齐信号稀疏和计算成本高而表现不佳。
• 通过识别并过滤句子级与词语级无关内容，提升长文本匹配的有效性与效率。
• 利用 PageRank 算法在基于 Transformer 的架构中系统性地检测并抑制句子与词语层面的噪声。
• 通过基于图的排序聚焦结构与语义上重要的组件，实现对长文本全局语义意义的更好捕捉。
• 在保持计算开销更低的前提下，超越短文本及近期长文本匹配模型的性能表现。

提出的方法

• 基于句子之间的词汇重叠（如 Jaccard 相似度）构建句子相似度图，再应用 PageRank 识别并过滤低重要性句子。
• 在 Transformer 的每个自注意力模块内，利用注意力得分作为边权重构建词语级图，形成每层动态变化的图结构。
• 在词语图上应用 PageRank 计算每个词语的重要性得分，实现在各层中逐步过滤低重要性词语。
• 将过滤过程集成到 Transformer 编码器中，先执行句子级过滤，再进行词语级过滤，从而降低计算负载。
• 引入可学习的缩减比例（α）以控制每层过滤的词语比例，实验表明 α=10% 时性能最优。
• 将过滤机制与标准自注意力机制结合，使模型能够端到端地联合学习过滤与匹配信号。

Figure 1. The top example is a short-form text matching for the paraphrasing identification, and the lines indicate the alignments between words from two sentences. The bottom example is a long-form text matching for the redundancy news identification, and the highlights indicate the important match

实验结果

研究问题

• RQ1 能否基于句子相似度图有效利用 PageRank 识别并过滤长文本匹配中的噪声句子？
• RQ2 能否通过从注意力得分构建动态词语图，并在分层方式下应用 PageRank，有效过滤词语级噪声？
• RQ3 与标准 Transformer 相比，句子与词语的分层过滤是否能提升长文本匹配任务的性能？
• RQ4 所提出的过滤机制对计算效率的影响如何，特别是在推理速度与训练时间方面？
• RQ5 在长文本匹配背景下，过滤比例与性能之间的最优平衡点是什么？

主要发现

• Match-Ignition 在七个公开的长文本匹配数据集上达到最先进性能，优于当前最先进短文本及近期长文本匹配模型。
• 在 CNSE 与 CNSS 数据集上，α=10% 的词语缩减比例模型表现最佳，表明在不损失准确率的前提下实现了最优噪声过滤。
• 当 α=20% 时，模型训练速度提升 1.6 倍，推理速度提升 2 倍（相比 α=0%），显著提升效率。
• 基于 PageRank 的词语过滤优于随机选择与嵌入范数策略，且比仅使用注意力权重更有效，证实了图传播机制的价值。
• 可视化结果显示，事件地点、政策名称等关键词获得更高重要性得分，与人工标注的相关性一致。
• 特殊标记如 [CLS] 和 [SEP] 被识别为高度重要，验证了其在长文本匹配中作为全局与结构标记的关键作用。

Figure 2. The overall architecture of Match-Ignition. (a) represents the sentence-level filter, (b) represents the outputs of the sentence-level filter, and (c) represents the word-level filter.