[Paper Review] Assessing Crosslingual Discourse Relations in Machine Translation

Karin Smith, Lucia Specia|arXiv (Cornell University)|Oct 7, 2018

Natural Language Processing Techniques20 references4 citations

TL;DR

This paper proposes a reference-free, source-based evaluation method for assessing crosslingual discourse relation transfer in machine translation using crosslingual discourse embeddings and discourse relation mapping. It outperforms standard metrics by measuring semantic preservation of discourse relations from source to target, achieving post-edited translations scored higher than MT outputs in 78% of cases, demonstrating its potential for realistic, discourse-aware MT evaluation.

ABSTRACT

In an attempt to improve overall translation quality, there has been an increasing focus on integrating more linguistic elements into Machine Translation (MT). While significant progress has been achieved, especially recently with neural models, automatically evaluating the output of such systems is still an open problem. Current practice in MT evaluation relies on a single reference translation, even though there are many ways of translating a particular text, and it tends to disregard higher level information such as discourse. We propose a novel approach that assesses the translated output based on the source text rather than the reference translation, and measures the extent to which the semantics of the discourse elements (discourse relations, in particular) in the source text are preserved in the MT output. The challenge is to detect the discourse relations in the source text and determine whether these relations are correctly transferred crosslingually to the target language -- without a reference translation. This methodology could be used independently for discourse-level evaluation, or as a component in other metrics, at a time where substantial amounts of MT are online and would benefit from evaluation where the source text serves as a benchmark.

Motivation & Objective

• To address the limitation of reference-based MT evaluation, which restricts assessment to single gold-standard translations and ignores discourse-level semantics.
• To develop a method that evaluates MT output based on semantic preservation of discourse relations in the source text, without relying on a reference translation.
• To enable large-scale, real-time evaluation of online MT by using the source text as a benchmark for discourse-level correctness.
• To improve MT evaluation by capturing crosslingual transfer of explicit discourse relations, particularly through lexical cues like discourse connectives.
• To explore the feasibility of using crosslingual discourse embeddings to model semantic equivalences across languages in a context-sensitive manner.

Proposed method

• The method detects discourse relations in the source text using a discourse parser and identifies discourse connectives as cues for specific relations (e.g., Contingency, Purpose).
• Crosslingual discourse embeddings are trained to capture semantic equivalences of multiword discourse connectives across French (source) and English (target) languages.
• A likelihood score from the crosslingual embeddings estimates the probability of correct discourse relation transfer from source to target language.
• A weighted score evaluates the correctness of the discourse relation transfer, combining embedding similarity and discourse relation mapping.
• The model compares MT output against the source text, not a reference, to assess whether the semantics of the discourse relation are preserved.
• The evaluation is applied across multiple MT systems (e.g., RBMT1, Stanford, UEdin), with scores validated against post-edited versions as a proxy for correctness.

Experimental results

Research questions

• RQ1 Can discourse relations in a source text be reliably detected and mapped to their target language counterparts without relying on a reference translation?
• RQ2 To what extent can crosslingual discourse embeddings capture semantic equivalences of discourse connectives across languages like French and English?
• RQ3 How well does the proposed method distinguish between MT output and post-edited versions in terms of discourse relation preservation?
• RQ4 Does the model reward MT systems that preserve discourse semantics even when they differ from human reference translations?
• RQ5 How does the model perform on implicit versus explicit discourse relations, and what are the limitations in handling intersentential relations?

Key findings

• The proposed method scored post-edited translations higher or equal to MT outputs in 78% of the evaluated documents, indicating its sensitivity to discourse-level improvements.
• Rule-based MT systems (e.g., RBMT1) performed better under the proposed metric than in standard evaluations, particularly due to correct rendering of discourse connectives like 'so that' for 'pour que'.
• Systems that failed to preserve key discourse markers—such as 'to enable' or 'so that'—were penalized, even if they were grammatically correct, highlighting the model’s focus on semantic fidelity.
• The model identified that STANFORD and UEDIN systems omitted the infinitival construction 'to enable' or 'to make', leading to loss of discourse relation semantics despite high fluency.
• The method demonstrated a higher correlation with human judgment (Spearman’s rho = 0.263) than standard metrics, suggesting improved alignment with discourse-level understanding.
• The approach revealed that human reference translations often implicitly encode discourse relations (e.g., via 'to enable'), which are not always preserved in MT, underscoring the need for source-based evaluation.