[Paper Review] Exact Hard Monotonic Attention for Character-Level Transduction

Shijie Wu, Ryan Cotterell|arXiv (Cornell University)|May 15, 2019

Natural Language Processing Techniques22 references4 citations

TL;DR

This paper proposes an exact hard monotonic attention mechanism for character-level sequence-to-sequence transduction, using dynamic programming to jointly learn monotonic alignments and transduction. It achieves state-of-the-art performance on morphological inflection and strong results on grapheme-to-phoneme and transliteration tasks by enforcing strict monotonicity during joint training, outperforming non-monotonic and pipeline-based baselines.

ABSTRACT

Many common character-level, string-to string transduction tasks, e.g., grapheme-tophoneme conversion and morphological inflection, consist almost exclusively of monotonic transductions. However, neural sequence-to sequence models that use non-monotonic soft attention often outperform popular monotonic models. In this work, we ask the following question: Is monotonicity really a helpful inductive bias for these tasks? We develop a hard attention sequence-to-sequence model that enforces strict monotonicity and learns a latent alignment jointly while learning to transduce. With the help of dynamic programming, we are able to compute the exact marginalization over all monotonic alignments. Our models achieve state-of-the-art performance on morphological inflection. Furthermore, we find strong performance on two other character-level transduction tasks. Code is available at https://github.com/shijie-wu/neural-transducer.

Motivation & Objective

• To investigate whether monotonicity is a beneficial inductive bias in character-level transduction tasks like morphological inflection and grapheme-to-phoneme conversion.
• To address the underperformance of prior monotonic models by jointly training alignments with the transduction model, rather than using pipeline approaches.
• To develop an exact inference algorithm for monotonic alignments that enables end-to-end training with dynamic programming.
• To evaluate whether enforcing strict monotonicity improves performance over non-monotonic or loosely constrained models.
• To determine whether increased model capacity (1st-order) improves performance over simpler 0th-order models under monotonic constraints.

Proposed method

• Extends the 0th-order hard attention model of Wu et al. (2018) to enforce strict monotonicity in alignments via a dynamic programming-based exact marginalization over all valid monotonic alignments.
• Introduces a 1st-order hard attention variant that models alignment transitions as Markovian, allowing more expressive alignment learning while preserving monotonicity.
• Uses dynamic programming to compute the exact log-likelihood of target sequences under monotonic alignment constraints in O(|x|·|y|·|Σ_y|) time.
• Employs a greedy decoding scheme for inference and a joint training objective that optimizes both alignment and transduction simultaneously.
• Implements a tractable computation of the marginal likelihood over monotonic alignments by reordering summation terms to avoid exponential complexity.
• Applies the model to three character-level transduction tasks: morphological inflection, grapheme-to-phoneme conversion, and named-entity transliteration.

Experimental results

Research questions

• RQ1 Does enforcing strict monotonicity in hard attention improve performance on character-level transduction tasks?
• RQ2 Is the underperformance of prior monotonic models due to the lack of joint training between alignment and transduction?
• RQ3 Can exact dynamic programming inference over monotonic alignments lead to better generalization than non-monotonic or pipeline systems?
• RQ4 Does increasing model capacity (1st-order vs 0th-order) improve performance under monotonic constraints?
• RQ5 Is monotonicity a useful inductive bias when alignment is jointly learned rather than precomputed?

Key findings

• The 1st-order monotonic hard attention model achieves state-of-the-art single-model performance on the CoNLL-SIGMORPHON 2017 morphological inflection shared task, with a 94.8% accuracy, surpassing prior best single models.
• The 0th-order monotonic model (0-mono) outperforms both the non-monotonic 0-hard and soft attention models on all three tasks, indicating that strict monotonicity is beneficial when alignment is jointly learned.
• The 1st-order monotonic model (1-mono) achieves the highest performance on grapheme-to-phoneme conversion (41.2% ACC) and named-entity transliteration (0.113 MLD), showing that increased expressiveness helps in some tasks.
• The 0-mono and 1-mono models tie at 94.8% on morphological inflection, suggesting that the additional parameters in 1st-order models do not always improve performance, but the monotonic constraint is key.
• The 0-mono model outperforms the 0-hard model across all tasks, demonstrating that joint learning of monotonic alignment is more effective than non-monotonic or pipeline approaches.
• The results indicate that the prior underperformance of monotonic models was likely due to the lack of joint training, not monotonicity itself.