[Paper Review] Solving Arithmetic Word Problems with Transformers and Preprocessing of Problem Text

Kaden Griffith, Jugal Kalita|arXiv (Cornell University)|Jun 2, 2021

Natural Language Processing Techniques21 references4 citations

TL;DR

This paper proposes a Transformer-based sequence-to-sequence model that translates arithmetic word problems into arithmetic expressions using prefix, postfix, or infix notation. By preprocessing problems with number tagging and linguistic filtering (e.g., LST, stopword removal), the model achieves state-of-the-art accuracy, improving by over 30% on some datasets compared to prior methods, with postfix representation yielding the best performance.

ABSTRACT

This paper outlines the use of Transformer networks trained to translate math word problems to equivalent arithmetic expressions in infix, prefix, and postfix notations. We compare results produced by many neural configurations and find that most configurations outperform previously reported approaches on three of four datasets with significant increases in accuracy of over 20 percentage points. The best neural approaches boost accuracy by 30% when compared to the previous state-of-the-art on some datasets.

Motivation & Objective

• To improve the accuracy of automatic arithmetic word problem solving using neural sequence-to-sequence models.
• To investigate whether preprocessing techniques like number tagging and linguistic filtering enhance model generalization and reduce spurious number generation.
• To evaluate the effectiveness of different arithmetic expression formats (infix, prefix, postfix) in neural translation for math word problems.
• To demonstrate that standard Transformer architectures, without external modules, can outperform specialized models on this task.
• To release code for reproducibility and future research in automated math problem solving.

Proposed method

• The model uses a Transformer encoder-decoder architecture to translate word problem text into arithmetic expressions in infix, prefix, or postfix notation.
• Numbers in word problems are replaced with generic tags (e.g., ⟨i⟩, ⟨q⟩, ⟨x⟩) during preprocessing to prevent model from generating spurious numbers not in the input.
• Linguistic preprocessing includes the use of Lemma-based Stopword Tagging (LST), which filters out irrelevant words and improves model focus on key mathematical cues.
• The model is trained end-to-end on four benchmark datasets using cross-entropy loss and beam search decoding.
• Postfix and prefix notations are preferred over infix due to their unambiguous structure, avoiding parentheses balancing issues.
• Preprocessing includes filtering stop words and applying part-of-speech tagging, though some combinations (e.g., lemmatization + stopword removal) degrade performance.

Experimental results

Research questions

• RQ1 Does preprocessing with number tagging and linguistic filtering improve the accuracy of Transformer-based solvers for arithmetic word problems?
• RQ2 Which arithmetic expression format—prefix, postfix, or infix—yields the highest translation accuracy in a neural sequence-to-sequence model?
• RQ3 Can a standard Transformer architecture outperform specialized neural architectures designed specifically for math word problem solving?
• RQ4 How do different preprocessing strategies (e.g., LST, stopword removal, lemmatization) affect model performance and generalization?
• RQ5 To what extent does the model avoid generating numbers not present in the original problem, and how does preprocessing mitigate this?

Key findings

• The Transformer model with postfix notation achieved the highest accuracy, outperforming previous state-of-the-art methods by over 30 percentage points on certain datasets.
• The use of number tagging with generic placeholders (⟨i⟩, ⟨q⟩, ⟨x⟩) significantly reduced the generation of spurious numbers not present in the input.
• Lemmatization combined with stopword removal led to a notable drop in performance, indicating that information loss from aggressive filtering harms model learning.
• Postfix and prefix notations improved model stability and accuracy compared to infix, which requires correct parentheses placement and increases decoding complexity.
• The LST preprocessing method (lemma-based stopword tagging) improved performance across all datasets and enhanced inference consistency.
• The model achieved an average accuracy improvement of nearly 10% over prior approaches, demonstrating that simple, well-designed preprocessing can significantly boost performance without architectural complexity.