[Paper Review] When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition

Bohan Li, Ye Yuan+6 more|arXiv (Cornell University)|Jul 23, 2022

Handwritten Text Recognition Techniques4 citations

TL;DR

This paper proposes Counting-Aware Network (CAN), an end-to-end framework that jointly optimizes handwritten mathematical expression recognition (HMER) and symbol counting using a weakly-supervised counting module. By leveraging LaTeX annotations without additional labeling, CAN improves attention accuracy and reduces recognition errors, achieving state-of-the-art performance on CROHME datasets with marginal computational overhead.

ABSTRACT

Recently, most handwritten mathematical expression recognition (HMER) methods adopt the encoder-decoder networks, which directly predict the markup sequences from formula images with the attention mechanism. However, such methods may fail to accurately read formulas with complicated structure or generate long markup sequences, as the attention results are often inaccurate due to the large variance of writing styles or spatial layouts. To alleviate this problem, we propose an unconventional network for HMER named Counting-Aware Network (CAN), which jointly optimizes two tasks: HMER and symbol counting. Specifically, we design a weakly-supervised counting module that can predict the number of each symbol class without the symbol-level position annotations, and then plug it into a typical attention-based encoder-decoder model for HMER. Experiments on the benchmark datasets for HMER validate that both joint optimization and counting results are beneficial for correcting the prediction errors of encoder-decoder models, and CAN consistently outperforms the state-of-the-art methods. In particular, compared with an encoder-decoder model for HMER, the extra time cost caused by the proposed counting module is marginal. The source code is available at https://github.com/LBH1024/CAN.

Motivation & Objective

• To address the instability of attention mechanisms in encoder-decoder models for HMER, especially in complex or long-formula scenarios.
• To explore the complementary relationship between symbol counting and HMER, proposing that counting can improve attention localization and recognition accuracy.
• To design a weakly-supervised counting module that uses only standard HMER annotations (LaTeX sequences), avoiding costly symbol-level bounding box annotations.
• To develop a unified, end-to-end trainable network that jointly optimizes HMER and symbol counting for improved robustness and generalization.

Proposed method

• Proposes a weakly-supervised counting module (MSCM) that predicts the number of each symbol class using only LaTeX sequences as supervision, without requiring position annotations.
• Integrates the MSCM into an attention-based encoder-decoder model (e.g., DWAP) to form the Counting-Aware Network (CAN), enabling joint optimization of HMER and counting.
• Uses the predicted counting map as a spatial-aware signal to guide the decoder’s attention, improving localization of symbols like subscripts and superscripts.
• Employs a context-aware decoder (CCAD) that leverages both HMER and counting signals to enhance attention accuracy during sequential decoding.
• Trains the entire network end-to-end with a multi-task loss combining HMER and counting objectives, using mean absolute error (MAE) and mean squared error (MSE) for counting supervision.
• Validates the method on CROHME 2014, 2016, and 2019 benchmarks, comparing against strong baselines including DWAP and ABM.

Experimental results

Research questions

• RQ1 Can symbol counting improve the attention accuracy of encoder-decoder models in HMER, especially in cases with complex layouts or long sequences?
• RQ2 How does weakly-supervised counting—using only LaTeX sequences—compare to fully-annotated counting in terms of HMER performance?
• RQ3 To what extent does joint optimization of HMER and counting improve both recognition accuracy and counting reliability?
• RQ4 Can the HMER task itself enhance the performance of the counting module through shared representation learning?
• RQ5 Does the proposed counting module introduce significant inference overhead, and is it practical for real-world deployment?

Key findings

• CAN achieves state-of-the-art performance on CROHME 2014, 2016, and 2019, with 57.26%, 56.15%, and 55.96% exact match rate when using ABM as the baseline.
• When using DWAP as the baseline, CAN achieves 57.00% (CROHME 2014), 56.06% (CROHME 2016), and 54.88% (CROHME 2019) exact match rate, outperforming the original model.
• Using ground-truth counting vectors, the exact match rate increases to 62.44% on CROHME 2014, demonstrating the strong impact of accurate counting supervision.
• The counting module reduces MAE and MSE by 31.25% and 15.91% respectively when HMER is jointly optimized, showing that HMER improves counting accuracy.
• The extra inference time cost of the counting module is marginal, confirming its practicality for real-time applications.
• Visualizations show that CAN correctly locates missed or redundant symbols (e.g., 'dy', '∑', 'i') that the baseline model fails to attend to, confirming improved attention localization.