[Paper Review] Investigating Math Word Problems using Pretrained Multilingual Language Models

Minghuan Tan, Lei Wang+2 more|arXiv (Cornell University)|May 19, 2021

Topic Modeling30 references4 citations

TL;DR

This paper investigates cross-lingual and multilingual math word problem (MWP) solving using pretrained multilingual language models, proposing a sequence-to-sequence model with copy mechanism over mBERT and XLM-R. Results show that MWP solvers are not transferable across languages even with shared operators and constants, but performance improves significantly when training and evaluation problems share similar problem types, highlighting domain similarity as a key factor over language similarity.

ABSTRACT

In this paper, we revisit math word problems~(MWPs) from the cross-lingual and multilingual perspective. We construct our MWP solvers over pretrained multilingual language models using sequence-to-sequence model with copy mechanism. We compare how the MWP solvers perform in cross-lingual and multilingual scenarios. To facilitate the comparison of cross-lingual performance, we first adapt the large-scale English dataset MathQA as a counterpart of the Chinese dataset Math23K. Then we extend several English datasets to bilingual datasets through machine translation plus human annotation. Our experiments show that the MWP solvers may not be transferred to a different language even if the target expressions have the same operator set and constants. But for both cross-lingual and multilingual cases, it can be better generalized if problem types exist on both source language and target language.

Motivation & Objective

• To study whether MWP solvers trained on one language can generalize to another language using multilingual pretraining.
• To evaluate whether combining multilingual datasets improves MWP solving performance across languages.
• To identify critical factors—such as language similarity, operator set, and problem type—that influence cross-lingual and multilingual MWP generalization.
• To construct and release new bilingual MWP datasets by translating and human-annotating existing English datasets into Chinese for comparative analysis.
• To empirically assess the effectiveness of contrastive learning and template-based training in improving zero-shot and few-shot MWP generalization.

Proposed method

• Fine-tune multilingual BERT (mBERT) and XLM-Roberta (XLM-R) on monolingual and multilingual MWP datasets using a sequence-to-sequence model with copy mechanism.
• Adapt the large-scale English MathQA dataset to match the operator set and expression format of the Chinese Math23K dataset for consistent comparison.
• Construct bilingual MWP datasets by translating MathQA and Math23K into Chinese using machine translation followed by human annotation for quality control.
• Apply template-based contrastive learning to improve zero-shot generalization by aligning semantically similar problem templates across languages.
• Train models using a batch size of 160 on a single 2080Ti GPU with a polynomial decay learning rate scheduler and early stopping after 30 epochs of no improvement.
• Evaluate models on monolingual test sets, zero-shot cross-lingual settings, and bilingual evaluation sets to measure transferability and multilingual generalization.

Experimental results

Research questions

• RQ1 Can a MWP solver trained on a monolingual dataset (e.g., Chinese Math23K) successfully solve problems in a different language (e.g., English) when using multilingual pretraining?
• RQ2 Does combining monolingual datasets from multiple languages into a single multilingual training set improve performance on each individual language compared to monolingual training?
• RQ3 To what extent does problem type similarity between source and target languages affect cross-lingual MWP solving performance, even when operators and constants are shared?
• RQ4 How does template-based contrastive learning affect zero-shot and few-shot MWP generalization across languages?
• RQ5 What are the key factors—language similarity, domain similarity, or operator set consistency—that most influence the success of multilingual MWP solvers?

Key findings

• A monolingual MWP solver trained on Chinese Math23K (mBERT-zh) achieves 76.3% accuracy on its own test set but drops to 49.5% when evaluated on English problems, indicating near-zero cross-lingual transferability despite shared operators and constants.
• A multilingual model trained on both Math23K and adapted MathQA (mBERT-xl) achieves 79.0% accuracy on English and 76.3% on Chinese, outperforming monolingual models on both languages and showing that multilingual training improves language-agnostic performance.
• Performance gains in multilingual training are most significant when problem types are similar across languages; for example, mBERT-xl improves by 3.1 points on Math23K and 2.4 points on MathQA compared to monolingual baselines.
• Template-based contrastive learning boosts performance on the test set by up to 3 points (e.g., 76.3% to 79.4% on Math23K), but harms zero-shot performance on English, likely due to template diversity mismatch between datasets.
• When constrained to share only templates present in both languages, the model sees a drop in test set performance but gains the most in zero-shot English evaluation, confirming that problem type alignment is more critical than language for generalization.
• The study demonstrates that domain similarity (problem types) is a stronger predictor of MWP solver generalization than language similarity, even when operator sets and constants are identical.