[Paper Review] Multi-Task Neural Models for Translating Between Styles Within and Across Languages

Xing Niu, Sudha Rao+1 more|arXiv (Cornell University)|Jun 12, 2018

Natural Language Processing Techniques36 references52 citations

TL;DR

The paper proposes a unified multi-task neural MT model that performs monolingual formality transfer in both directions and formality-aware translation from French to English, achieving state-of-the-art results for formality transfer and competitive results for formality-sensitive MT without style-annotated translation data.

ABSTRACT

Generating natural language requires conveying content in an appropriate style. We explore two related tasks on generating text of varying formality: monolingual formality transfer and formality-sensitive machine translation. We propose to solve these tasks jointly using multi-task learning, and show that our models achieve state-of-the-art performance for formality transfer and are able to perform formality-sensitive translation without being explicitly trained on style-annotated translation examples.

Motivation & Objective

• Motivate and address formality as a core stylistic dimension in text generation.
• Develop a single neural MT model that can perform bidirectional formality transfer in English.
• Enable formality-controlled translation (FSMT) from French to English using side constraints.
• Demonstrate that joint multi-task learning leverages both monolingual style-parallel data and bilingual data to improve FT and FSMT.
• Evaluate with automatic BLEU and human judgments to assess formality control, fluency, and meaning preservation.

Proposed method

• Use an attentional seq2seq NMT architecture with shared encoders/decoders.
• Bi-directional English formality transfer by concatenating dual-direction data and prepending a target-formality tag (<F> or <I>) to the source sentence.
• Formality-sensitive MT with side constraints by tagging bilingual French-English data with formal or informal target style, selected via cross-entropy difference (CED) to approximate formality.
• Multi-task learning setup that jointly trains on both FT and FSMT with several configurations (MultiTask-tag-style, MultiTask-style, MultiTask-random).
• Preprocessing includes normalization, tokenization, true-casing, joint BPE (32k), and tied embeddings; model details include Sockeye-based attentional encoder-decoder, LSTM encoder (512), attention (512), dropout, Adam optimization, and ensemble decoding.

Experimental results

Research questions

• RQ1 Can a single model perform formality transfer in both directions effectively?
• RQ2 How should monolingual FT data be combined with bilingual FSMT data to maximize performance?
• RQ3 What kinds of bilingual data (and tagging strategies) are most beneficial for joint FT and FSMT learning?
• RQ4 Can FSMT be learned without explicit style-annotated translation data through multi-task learning?

Key findings

• Bi-directional formality transfer with multi-task learning achieves state-of-the-art BLEU on FT (72.13 and 75.37 in the two transfer directions with augmentation and ensembling).
• In FSMT, neural models with multi-task learning show competitive performance and better formality control than some baselines, e.g., MultiTask-random achieving 25.24 and 25.14 BLEU for FR→Formal-EN and FR→Informal-EN, respectively, and outperforming several non-neural baselines in human evaluations.
• Human evaluation shows MultiTask-tag-style yields significantly more informal output when transferring formal→informal and slightly more formal when transferring informal→formal, with comparable meaning preservation to strong baselines.
• The joint model can learn FSMT without explicit style-annotated translation data, and multi-task learning improves FT beyond prior single-task approaches.
• Bilingual data size and data selection (CED-based tagging) positively impact FT, with a tipping point after which gains plateau, while translation quality continues to improve with more data.