[Paper Review] A Meaning-based Statistical English Math Word Problem Solver

Chao-Chun Liang, Yu‐Shiang Wong+2 more|arXiv (Cornell University)|Mar 16, 2018

Natural Language Processing Techniques23 references3 citations

TL;DR

This paper presents MeSys, a meaning-based statistical approach for solving English math word problems by converting text into logic forms with context-aware role-tags, enabling accurate operator and operand selection via statistical models. It outperforms prior systems on standard and noisy datasets, demonstrating superior semantic understanding over pattern matching.

ABSTRACT

We introduce MeSys, a meaning-based approach, for solving English math word problems (MWPs) via understanding and reasoning in this paper. It first analyzes the text, transforms both body and question parts into their corresponding logic forms, and then performs inference on them. The associated context of each quantity is represented with proposed role-tags (e.g., nsubj, verb, etc.), which provides the flexibility for annotating an extracted math quantity with its associated context information (i.e., the physical meaning of this quantity). Statistical models are proposed to select the operator and operands. A noisy dataset is designed to assess if a solver solves MWPs mainly via understanding or mechanical pattern matching. Experimental results show that our approach outperforms existing systems on both benchmark datasets and the noisy dataset, which demonstrates that the proposed approach understands the meaning of each quantity in the text more.

Motivation & Objective

• To develop a math word problem solver that understands semantic meaning rather than relying on surface-level patterns.
• To model the physical meaning of quantities using role-tags (e.g., nsubj, verb) for context-aware reasoning.
• To improve operator and operand selection through statistical models trained on meaning-structured logic forms.
• To evaluate solvers on a noisy dataset to distinguish between true understanding and mechanical matching.
• To achieve state-of-the-art performance on standard and adversarial math word problem benchmarks.

Proposed method

• Transforming the problem body and question into logic forms using syntactic and semantic analysis.
• Annotating each quantity with role-tags (e.g., nsubj, verb) to encode its physical meaning and contextual role.
• Using statistical models to predict the correct operator and operands based on the logic forms and role-tagged context.
• Designing a noisy dataset to test whether the solver relies on semantic understanding or pattern matching.
• Applying inference on the logic forms to derive the final answer using the selected operations.
• Training and evaluating the system on benchmark datasets and the proposed noisy dataset to measure robustness and understanding.

Experimental results

Research questions

• RQ1 Can a statistical model effectively capture the semantic meaning of quantities in math word problems through role-tagged context?
• RQ2 Does the proposed method outperform existing systems in solving standard and adversarial math word problems?
• RQ3 To what extent does the system rely on semantic understanding versus surface-level pattern matching?
• RQ4 How well do role-tags improve the accuracy of operator and operand selection in math word problem solving?
• RQ5 Can a meaning-based approach generalize to noisy or perturbed problem formulations?

Key findings

• MeSys achieves state-of-the-art performance on standard benchmark datasets, outperforming previous systems.
• The model significantly outperforms existing approaches on the proposed noisy dataset, indicating stronger semantic understanding.
• The use of role-tags enhances context representation, improving the accuracy of operand and operator selection.
• The noisy dataset effectively distinguishes systems that rely on pattern matching from those that understand meaning.
• Statistical modeling of logic forms with role-tagged context leads to more robust and interpretable reasoning in math word problems.
• The results confirm that MeSys solves problems through semantic reasoning rather than mechanical pattern matching.