[Paper Review] An Exploration of Neural Sequence-to-Sequence Architectures for Automatic Post-Editing

Marcin Junczys-Dowmunt, Roman Grundkiewicz|arXiv (Cornell University)|Jun 13, 2017

Natural Language Processing Techniques14 references15 citations

TL;DR

This paper proposes dual-attention neural sequence-to-sequence models for automatic post-editing (APE) that jointly encode both source (src) and machine translation (mt) inputs to directly predict post-edited (pe) output. It demonstrates that end-to-end dual-attention models incorporating all available data achieve state-of-the-art results on the WMT-2016 APE benchmark, outperforming prior systems by over 1.5 TER points, while hard-attention variants achieve competitive performance with fewer edits.

ABSTRACT

In this work, we explore multiple neural architectures adapted for the task of automatic post-editing of machine translation output. We focus on neural end-to-end models that combine both inputs $mt$ (raw MT output) and $src$ (source language input) in a single neural architecture, modeling $\{mt, src\} ightarrow pe$ directly. Apart from that, we investigate the influence of hard-attention models which seem to be well-suited for monolingual tasks, as well as combinations of both ideas. We report results on data sets provided during the WMT-2016 shared task on automatic post-editing and can demonstrate that dual-attention models that incorporate all available data in the APE scenario in a single model improve on the best shared task system and on all other published results after the shared task. Dual-attention models that are combined with hard attention remain competitive despite applying fewer changes to the input.

Motivation & Objective

• To investigate end-to-end neural sequence-to-sequence architectures that jointly model both source (src) and machine translation (mt) inputs for automatic post-editing.
• To evaluate the effectiveness of hard-attention mechanisms in monolingual APE tasks.
• To compare dual-soft-attention, hard-attention, and hybrid attention mechanisms in improving APE performance.
• To demonstrate that integrating full {src, mt, pe} triplets into a single end-to-end model yields superior results compared to prior systems.

Proposed method

• Proposes a dual-attention encoder-decoder architecture that jointly encodes both src and mt inputs to predict pe output in a single end-to-end model.
• Employs conditional GRUs with soft attention for standard attention modeling, allowing the decoder to focus on relevant parts of both input sequences.
• Introduces hard-attention mechanisms that enforce monotonic, non-differentiable attention over input sequences, reducing the number of edits.
• Combines hard-attention with soft-attention and dual-soft-attention mechanisms to explore hybrid attention strategies.
• Uses the Marian neural machine translation toolkit for implementation and training.
• Applies Minimum Error Rate Training (MERT) to tune model components and optimize for TER metric.

Experimental results

Research questions

• RQ1 Can dual-attention models that jointly encode both src and mt inputs outperform single-source models in APE?
• RQ2 How do hard-attention mechanisms compare to soft-attention in terms of edit efficiency and performance on APE?
• RQ3 Do hybrid architectures combining hard and soft attention improve APE performance while maintaining faithfulness to the original MT output?
• RQ4 Can end-to-end models trained on full {src, mt, pe} triplets achieve state-of-the-art results on the WMT-2016 APE benchmark?

Key findings

• Dual-attention models that incorporate both src and mt inputs in a single end-to-end architecture achieve a TER score of 1.52 points lower than the best WMT-2016 shared task system.
• Ensembles of dual-attention models improve performance by 0.7 TER points over the best previously published system combination on the same test set.
• Hard-attention models achieve competitive results with significantly fewer edits—demonstrating higher faithfulness to the original MT output.
• The dual-soft-attention model with hard-attention integration (M-CGRU-HARD) successfully recovers missing words like 'Satz' and adds grammatically correct particles such as 'aus', which monolingual models fail to do.
• Visualization shows that dual-attention models effectively use src to restore missing translations, while attention over mt remains monotonic.
• Ensembles reduce the number of deteriorated sentences but do not significantly increase the number of improved sentences, indicating that performance gains come from consistency rather than aggressive correction.