[Paper Review] Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition

Jianshu Zhang, Jun Du+1 more|arXiv (Cornell University)|Jan 5, 2018

Handwritten Text Recognition Techniques25 references20 citations

TL;DR

This paper proposes a multi-scale attention mechanism with a dense encoder for handwritten mathematical expression recognition, improving feature extraction and preserving fine-grained details lost in pooling operations. By integrating DenseNet for enhanced gradient flow and multi-scale attention to attend to both high- and low-resolution features, the model achieves state-of-the-art accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016 using only official training data.

ABSTRACT

Handwritten mathematical expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, we utilize the attention based encoder-decoder model that recognizes mathematical expression images from two-dimensional layouts to one-dimensional LaTeX strings. We improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and save the fine-grained details that will be dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-the-art methods with an expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset.

Motivation & Objective

• To address the challenges of complex 2D structures, handwriting ambiguity, and variable symbol scales in handwritten mathematical expression recognition.
• To overcome limitations of conventional methods that rely on predefined grammars and complex parsing algorithms.
• To improve feature representation and preserve fine-grained details lost during pooling in CNN-based encoders.
• To develop an end-to-end trainable, data-driven model that eliminates the need for explicit symbol segmentation.

Proposed method

• Adopts a densely connected convolutional network (DenseNet) as the encoder to strengthen feature extraction and improve gradient flow, especially on small datasets.
• Introduces a multi-scale dense encoder that generates both low-resolution (semantic-rich) and high-resolution (detail-preserving) feature maps.
• Designs a multi-scale attention mechanism that allows the decoder to attend to features from both high- and low-resolution branches during sequence generation.
• Uses an attention-based encoder-decoder framework where the decoder generates LaTeX strings token-by-token, attending to relevant image regions via multi-scale features.
• Employs skip connections in DenseNet to concatenate feature maps from all preceding layers, enhancing feature reuse and representation learning.
• Trains the model end-to-end using cross-entropy loss on image-to-LaTeX sequence mapping, with beam search for inference.

Experimental results

Research questions

• RQ1 Can a densely connected encoder improve feature representation and recognition accuracy in handwritten mathematical expression recognition compared to standard CNNs?
• RQ2 Does incorporating multi-scale attention help recover fine-grained visual details—especially for small symbols like decimal points—lost during pooling operations?
• RQ3 How does the proposed method perform on standard benchmarks like CROHME 2014 and CROHME 2016 compared to state-of-the-art approaches using only official training data?
• RQ4 To what extent does the multi-scale attention mechanism reduce under-parsing errors for small or ambiguous symbols?
• RQ5 Can the model generalize well across different datasets without relying on external or private training data?

Key findings

• The proposed model achieves a 52.8% expression recognition accuracy on the CROHME 2014 test set, outperforming all other systems using only official training data.
• On the CROHME 2016 benchmark, the model achieves 50.1% accuracy, surpassing the second-best system (Wiris at 49.6%) and demonstrating strong generalization.
• The multi-scale attention mechanism successfully recovers small symbols such as decimal points and numerals like '7' that are often missed by single-scale attention models.
• Visualization results confirm that high-resolution attention branches detect fine-grained details—such as the decimal point in '3.14'—that are lost in low-resolution feature maps.
• The dense encoder improves performance by enhancing feature learning and gradient flow, particularly beneficial on limited training data.
• The model achieves a 72.7% accuracy for predictions within one symbol error on CROHME 2014, indicating strong robustness and potential for further improvement.