[论文解读] A Joint Probabilistic Classification Model of Relevant and Irrelevant Sentences in Mathematical Word Problems

Suleyman Cetintas, Luo Si+4 more|arXiv (Cornell University)|Dec 1, 2010

Topic Modeling参考文献 24被引用 4

一句话总结

本文提出一种联合概率分类模型，通过建模所有句子之间的相关性以及问题与其它句子之间的相关性，识别数学应用题中的相关句与无关句。实验表明，该模型优于独立SVM分类与相关性感知SVM，证明联合建模可提升分类准确率，且去除停用词在此任务中反而适得其反。

ABSTRACT

Estimating the difficulty level of math word problems is an important task for many educational applications. Identification of relevant and irrelevant sentences in math word problems is an important step for calculating the difficulty levels of such problems. This paper addresses a novel application of text categorization to identify two types of sentences in mathematical word problems, namely relevant and irrelevant sentences. A novel joint probabilistic classification model is proposed to estimate the joint probability of classification decisions for all sentences of a math word problem by utilizing the correlation among all sentences along with the correlation between the question sentence and other sentences, and sentence text. The proposed model is compared with i) a SVM classifier which makes independent classification decisions for individual sentences by only using the sentence text and ii) a novel SVM classifier that considers the correlation between the question sentence and other sentences along with the sentence text. An extensive set of experiments demonstrates the effectiveness of the joint probabilistic classification model for identifying relevant and irrelevant sentences as well as the novel SVM classifier that utilizes the correlation between the question sentence and other sentences. Furthermore, empirical results and analysis show that i) it is highly beneficial not to remove stopwords and ii) utilizing part of speech tagging does not make a significant improvement although it has been shown to be effective for the related task of math word problem type classification.

研究动机与目标

• 提升数学应用题中句子相关性或无关性分类的准确性。
• 建模应用题中所有句子之间的依赖关系，包括问题与其他句子之间的关系。
• 评估在仅使用文本特征的基础上，引入句子相关性是否能提升分类性能。
• 探究预处理策略（如停用词去除与词性标注）对分类效果的影响。

提出的方法

• 开发一种联合概率分类模型，用于估计问题中所有句子分类决策的联合概率。
• 利用句子之间的相关性以及问题与其他句子之间的关系，提升分类的一致性。
• 以句子文本和结构关系（如问题-句子链接）作为输入特征。
• 与两种基线模型进行对比：独立SVM分类器与利用问题-句子关系的相关性感知SVM。
• 采用概率推理方法，在单一优化步骤中联合预测所有句子的相关性标签。
• 特征工程包括原始文本、词性标注与词汇特征，并对预处理策略进行消融研究。

实验结果

研究问题

• RQ1 建模句子间相关性是否能提升数学应用题中相关句与无关句的分类效果？
• RQ2 将问题与其它句子之间的关系纳入模型是否能提升分类性能？
• RQ3 在数学应用题中，去除停用词是否有利于句子相关性分类？
• RQ4 词性标注在此任务中是否能显著提升分类准确率？
• RQ5 联合概率模型相较于独立SVM与相关性感知SVM基线模型表现如何？

主要发现

• 联合概率模型在分类相关句与无关句方面显著优于独立SVM与相关性感知SVM。
• 通过利用所有句子之间的依赖关系以及问题与其他句子之间的关系，该模型实现了更高的F1分数。
• 实证结果表明，去除停用词会损害分类性能，说明保留停用词对此任务有益。
• 尽管词性标注在相关NLP任务（如数学应用题类型分类）中已被证明有效，但在此任务中并未带来显著性能提升。
• 相关性感知SVM优于独立SVM，证实问题与其他句子之间的结构关系具有信息价值。
• 联合模型的性能增益归因于其利用全局依赖关系，强制所有句子分类结果保持一致的能力。