[Paper Review] Linear programming word problems formulation using EnsembleCRF NER labeler and T5 text generator with data augmentations

JiangLong He, N Mamatha+3 more|arXiv (Cornell University)|Dec 30, 2022

Software Engineering Research4 citations

TL;DR

This paper proposes an ensemble approach combining EnsembleCRF for named entity recognition and a multi-task T5 text generator with data augmentation to automate the formulation of linear programming word problems. By applying LwTR, SR, MR, and SiS data augmentation and using entity wrappers to improve model input, the method achieves 83.5% declaration-level accuracy, outperforming baseline models and securing 5th place in the NL4Opt competition.

ABSTRACT

We propose an ensemble approach to predict the labels in linear programming word problems. The entity identification and the meaning representation are two types of tasks to be solved in the NL4Opt competition. We propose the ensembleCRF method to identify the named entities for the first task. We found that single models didn't improve for the given task in our analysis. A set of prediction models predict the entities. The generated results are combined to form a consensus result in the ensembleCRF method. We present an ensemble text generator to produce the representation sentences for the second task. We thought of dividing the problem into multiple small tasks due to the overflow in the output. A single model generates different representations based on the prompt. All the generated text is combined to form an ensemble and produce a mathematical meaning of a linear programming problem.

Motivation & Objective

• To address the challenge of automated formulation of linear programming word problems from natural language, especially for non-experts.
• To improve named entity recognition (NER) in low-resource linear programming word problems using ensemble modeling.
• To generate accurate mathematical meaning representations by combining multiple prompt-based T5 outputs.
• To overcome data scarcity in training by applying four data augmentation techniques: LwTR, SR, MR, and SiS.
• To enhance model performance by introducing entity wrappers to improve input sequence clarity and label consistency.

Proposed method

• Employed an ensemble of CRF-based models (EnsembleCRF) to combine predictions from multiple single models, improving label consistency and reducing error variance.
• Used a T5-based text generator with multiple prompt templates to generate diverse meaning representations, then ensembled the outputs for final mathematical formulation.
• Applied four data augmentation techniques—Label-wise token replacement (LwTR), Synonym replacement (SR), Mention replacement (MR), and Shuffle within segments (SiS)—to increase training data diversity and size.
• Introduced entity wrappers (e.g., <OBJ_NAME> for variables) to explicitly mark entities in input text, improving model attention and prediction accuracy.
• Optimized model hyperparameters including max_seq_length (600), max_output_length (750), and training_epochs (50) to handle long sequences and improve performance.
• Conducted ablation studies to evaluate the impact of data augmentation, multi-task learning, and entity wrapping on model performance.

Experimental results

Research questions

• RQ1 Can an ensemble of CRF models improve named entity recognition performance in low-resource linear programming word problems compared to single models?
• RQ2 How effective are data augmentation techniques (LwTR, SR, MR, SiS) in improving model generalization when training data is limited?
• RQ3 Does the use of entity wrappers in input sequences enhance the model's ability to correctly identify and represent variables and parameters in linear programming problems?
• RQ4 To what extent does multi-task prompting with a single T5 model improve the quality and consistency of mathematical meaning representation generation?
• RQ5 Why do model outputs sometimes fail to retrieve all instances of the same entity, especially when multiple values exist, and how can this be mitigated?

Key findings

• The ensembleCRF method significantly improved named entity recognition performance over single models, achieving a declaration-level accuracy of 83.5% when combining augmented input and multi-task training.
• Data augmentation using LwTR, SR, MR, and SiS increased training data diversity and contributed to a 12.7 percentage point improvement in accuracy compared to the original training setup (from 75.9% to 83.5%).
• The addition of entity wrappers increased the input sequence length to 590 tokens, exceeding the 512-token limit, requiring model adjustment to max_seq_length=600, which slightly reduced performance but enabled better entity localization.
• The multi-task prompting strategy with different prompts improved output diversity and consistency, though retrieval of multiple instances of the same entity remained a challenge.
• The final model achieved 83.5% declaration-level accuracy on the development set and ranked 5th in the NL4Opt competition with an F1 score of 0.780 on the test set, outperforming the baseline BART model (F1=0.608).
• Despite improvements, the model struggled to retrieve all instances of the same entity when multiple values were present, indicating a need for better handling of multi-instance entity detection in future work.