[Paper Review] Syntax-Aware Network for Handwritten Mathematical Expression Recognition

Ye Yuan, Xiao Liu+5 more|arXiv (Cornell University)|Mar 3, 2022

Handwritten Text Recognition Techniques5 citations

TL;DR

This paper proposes a syntax-aware network (SAN) for handwritten mathematical expression recognition (HMER) that integrates grammar rules and syntax-aware attention into an encoder-decoder framework. By modeling LaTeX sequence generation as a tree traversal process, SAN improves structural accuracy and reduces prediction errors from complex layouts or poor handwriting, achieving state-of-the-art performance on CROHME and a new large-scale dataset, HME100K.

ABSTRACT

Handwritten mathematical expression recognition (HMER) is a challenging task that has many potential applications. Recent methods for HMER have achieved outstanding performance with an encoder-decoder architecture. However, these methods adhere to the paradigm that the prediction is made "from one character to another", which inevitably yields prediction errors due to the complicated structures of mathematical expressions or crabbed handwritings. In this paper, we propose a simple and efficient method for HMER, which is the first to incorporate syntax information into an encoder-decoder network. Specifically, we present a set of grammar rules for converting the LaTeX markup sequence of each expression into a parsing tree; then, we model the markup sequence prediction as a tree traverse process with a deep neural network. In this way, the proposed method can effectively describe the syntax context of expressions, alleviating the structure prediction errors of HMER. Experiments on three benchmark datasets demonstrate that our method achieves better recognition performance than prior arts. To further validate the effectiveness of our method, we create a large-scale dataset consisting of 100k handwritten mathematical expression images acquired from ten thousand writers. The source code, new dataset, and pre-trained models of this work will be publicly available.

Motivation & Objective

• Address the challenge of structural prediction errors in handwritten mathematical expression recognition (HMER) due to complex 2D layouts and illegible handwriting.
• Overcome the limitations of standard image-to-sequence models that predict symbols sequentially without considering syntactic relationships.
• Develop a method that explicitly models the hierarchical syntax structure of mathematical expressions using grammar rules and neural attention.
• Create a large-scale, diverse HME100K dataset to support robust evaluation and future research in real-world HMER applications.
• Improve both recognition accuracy and inference efficiency by integrating syntactic constraints into deep learning architectures.

Proposed method

• Define a custom grammar to decompose LaTeX markup sequences into a syntax tree, enabling structured representation of mathematical expressions.
• Design a syntax-aware decoder that traverses the grammar tree component-by-component instead of symbol-by-symbol, preserving syntactic context.
• Introduce syntax-aware attention mechanism that focuses on relevant components based on tree structure, reducing attention on irrelevant or erroneous symbols.
• Use a fully convolutional encoder to extract visual features from handwritten expression images, followed by a decoder with syntax-aware attention for sequence generation.
• Train the end-to-end model using cross-entropy loss on LaTeX sequences, with beam search for inference to improve generation quality.
• Incorporate coverage attention as a baseline for comparison, demonstrating the superiority of syntax-aware attention in focusing on correct structural components.

Experimental results

Research questions

• RQ1 Can integrating explicit grammar rules into a neural encoder-decoder framework improve the structural accuracy of HMER models?
• RQ2 Does modeling sequence generation as a tree traversal process, rather than a character-by-character prediction, lead to better performance on complex mathematical expressions?
• RQ3 How does syntax-aware attention compare to standard coverage attention in focusing on correct components during HMER inference?
• RQ4 To what extent does the proposed method generalize across datasets of varying size and complexity, including real-world handwritten data?
• RQ5 Can a large-scale, diverse dataset like HME100K serve as a more robust benchmark for evaluating HMER systems beyond existing benchmarks?

Key findings

• The proposed syntax-aware network (SAN) achieves a 56.2% expression recognition accuracy on CROHME 2014, outperforming prior state-of-the-art methods.
• On the more challenging CROHME 2016 dataset, SAN achieves 53.6% accuracy, showing a 5.1% improvement over the baseline when using syntax-aware attention.
• On the newly introduced HME100K dataset, SAN achieves 67.1% recognition accuracy, demonstrating strong generalization on larger, more complex data.
• The ablation study confirms that grammar syntax integration improves accuracy by 6.2% on CROHME 2014, while syntax-aware attention adds an additional 7.1% gain.
• SAN is 3.5× faster than DWAP-TD and 2.6% faster than DWAP on the HME100K test set, indicating high inference efficiency.
• Qualitative analysis shows that syntax-aware attention correctly focuses on structural components, while coverage attention often mispredicts redundant or incorrect subtrees.