[Paper Review] Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

Verna Dankers, Christopher G. Lucas+1 more|arXiv (Cornell University)|May 30, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper investigates why Transformer-based neural machine translation (NMT) models frequently produce overly literal translations of idioms, despite their non-compositional meanings. By analyzing attention patterns and hidden representations in encoder and decoder heads across seven European languages, the authors find that figurative idioms are processed as unified lexical units with reduced context interaction, yet the model still tends to generate compositional outputs. The key contribution is identifying that the model's compositional inductive bias undermines non-compositional processing, suggesting architectural modifications are needed to improve idiom translation.

ABSTRACT

Unlike literal expressions, idioms' meanings do not directly follow from their parts, posing a challenge for neural machine translation (NMT). NMT models are often unable to translate idioms accurately and over-generate compositional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language. When Transformer emits a non-literal translation - i.e. identifies the expression as idiomatic - the encoder processes idioms more strongly as single lexical units compared to literal expressions. This manifests in idioms' parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder's cross-attention, figurative inputs result in reduced attention on source-side tokens. These results suggest that Transformer's tendency to process idioms as compositional expressions contributes to literal translations of idioms.

Motivation & Objective

• To investigate why Transformer-based NMT models frequently produce literal translations of idioms despite their non-compositional meanings.
• To analyze how idioms are represented in the hidden states and attention mechanisms of the Transformer encoder and decoder.
• To determine whether the model's compositional inductive bias impairs accurate idiom translation across multiple European languages.
• To evaluate whether intervention in hidden representations can causally alter the model's tendency toward compositional translations.
• To provide insights for architectural improvements that enhance non-compositional processing of idioms in NMT.

Proposed method

• The authors use a monolingual English corpus and heuristically label model translations as figurative or literal based on paraphrasing behavior.
• They apply interpretability techniques to compare self-attention in the encoder and cross-attention in the decoder between figurative and literal idiom expressions (PIEs).
• Hidden representations of idiom tokens are probed using INLP (Interpretable Nonlinear Probing) to assess whether figurativeness is linearly encoded in the model's representations.
• The authors intervene in the hidden states of the encoder across multiple layers to test the causal impact of representation changes on translation output.
• Human evaluation verifies the heuristic labeling method, ensuring that paraphrased translations are indeed figurative.
• Results are analyzed across seven European target languages to assess cross-lingual consistency of findings.

Experimental results

Research questions

• RQ1 How do attention patterns in the Transformer encoder differ between figurative and literal idiom expressions?
• RQ2 To what extent does the decoder rely on encoder outputs when translating figurative versus literal idioms?
• RQ3 Is the figurative meaning of idioms linearly encoded in the model's hidden representations, and can it be manipulated via intervention?
• RQ4 How does the model’s compositional inductive bias affect its ability to produce non-compositional translations of idioms?
• RQ5 Can architectural interventions that alter hidden representations reduce the model’s tendency toward literal translations of idioms?

Key findings

• Figurative idioms are processed as single lexical units in the encoder, with increased internal attention and reduced interaction with surrounding context.
• The decoder exhibits reduced cross-attention to the encoder’s output when translating figurative idioms, indicating greater independence in meaning derivation.
• Intervening in the hidden states using INLP successfully reduced paraphrasing and increased compositional translations, with success rates ranging from 27% (Swedish) to 40% (Spanish).
• The model’s tendency toward literal translations is robust, as over 70% of idiom translations remained word-for-word even after intervention, indicating weak non-compositional processing.
• Hidden representations of figurativeness are more detectable in higher layers, suggesting that non-compositional processing emerges later in the network.
• The results suggest that compositionality-favoring training techniques may harm idiom translation, warranting caution in future NMT design.