[论文解读] Sequence Prediction with Neural Segmental Models

Hao Tang|arXiv (Cornell University)|Sep 5, 2017

Natural Language Processing Techniques参考文献 108被引用 4

一句话总结

本文提出判别性分段级联（discriminative segmental cascades），一种多轮推理框架，通过引入高阶和神经分段特征，在保持高效解码的同时提升序列预测的准确性。此外，本文还提出使用边缘对数损失（marginal log loss）进行端到端训练，无需昂贵的手动对齐，统一了分段模型与联结主义时间分类（connectionist temporal classification, CTC）及隐马尔可夫模型（HMMs）。

ABSTRACT

Segments that span contiguous parts of inputs, such as phonemes in speech, named entities in sentences, actions in videos, occur frequently in sequence prediction problems. Segmental models, a class of models that explicitly hypothesizes segments, have allowed the exploration of rich segment features for sequence prediction. However, segmental models suffer from slow decoding, hampering the use of computationally expensive features. In this thesis, we introduce discriminative segmental cascades, a multi-pass inference framework that allows us to improve accuracy by adding higher-order features and neural segmental features while maintaining efficiency. We also show that instead of including more features to obtain better accuracy, segmental cascades can be used to speed up training and decoding. Segmental models, similarly to conventional speech recognizers, are typically trained in multiple stages. In the first stage, a frame classifier is trained with manual alignments, and then in the second stage, segmental models are trained with manual alignments and the out- puts of the frame classifier. However, obtaining manual alignments are time-consuming and expensive. We explore end-to-end training for segmental models with various loss functions, and show how end-to-end training with marginal log loss can eliminate the need for detailed manual alignments. We draw the connections between the marginal log loss and a popular end-to-end training approach called connectionist temporal classification. We present a unifying framework for various end-to-end graph search-based models, such as hidden Markov models, connectionist temporal classification, and segmental models. Finally, we discuss possible extensions of segmental models to large-vocabulary sequence prediction tasks.

研究动机与目标

• 解决分段模型因计算成本高昂的特征导致解码效率低下的问题。
• 降低训练分段模型时对耗时手动对齐的依赖。
• 在不牺牲速度的前提下，通过整合高阶和神经特征提升序列预测的准确性。
• 将各种端到端序列建模方法（包括HMMs、CTC和分段模型）统一到一个共同框架下。
• 将分段模型扩展至大规模词汇量的序列预测任务。

提出的方法

• 提出判别性分段级联：一种多轮推理机制，通过迭代方式利用高阶和神经特征优化预测结果。
• 引入使用边缘对数损失的端到端训练方法，避免对详细手动对齐的需求。
• 建立边缘对数损失与联结主义时间分类（CTC）之间的理论联系，证明在特定条件下二者等价。
• 设计一种基于图搜索的统一框架，将HMMs、CTC和分段模型作为特例纳入其中。
• 在两阶段训练过程中，初始使用帧分类器输出作为假设，后续由端到端学习替代。
• 利用神经网络建模分段级表示，实现在分段框架内更丰富的特征学习。

实验结果

研究问题

• RQ1 在使用复杂特征时，能否在保持高效解码的同时提升分段模型的准确性？
• RQ2 使用边缘对数损失的端到端训练能否消除分段模型对手动对齐的需求？
• RQ3 边缘对数损失与现有端到端方法（如CTC）之间有何关系？
• RQ4 能否开发一个统一框架，将HMMs、CTC和分段模型统一到相同的推理与训练范式下？
• RQ5 分段模型能否有效扩展至大规模词汇量的序列预测任务？

主要发现

• 判别性分段级联通过引入高阶和神经特征，在保持解码效率的同时，实现了比标准分段模型更高的准确性。
• 使用边缘对数损失的端到端训练消除了对手动对齐的需求，显著降低了标注成本。
• 边缘对数损失在理论上与CTC相关联，表明CTC可被视为所提框架的一个特例。
• 统一框架成功地将HMMs、CTC和分段模型整合到单一基于图搜索的训练与解码机制中。
• 所提方法使分段模型能够有效应用于大规模词汇量的序列预测任务，此前因复杂性而难以实现。
• 该方法在提升性能的同时保持了具有竞争力的推理速度，展示了在实际应用中的可行性。