[Paper Review] Towards Explainable Evaluation Metrics for Machine Translation

Christoph Leiter, Piyawat Lertvittayakumjorn+4 more|arXiv (Cornell University)|Jun 22, 2023

Topic Modeling6 citations

TL;DR

This paper proposes a framework for explainable evaluation metrics in machine translation, advocating for transparency to overcome distrust in black-box metrics like BERTScore and COMET. It surveys explainability techniques—feature attribution, perturbation analysis, and LLM-based explanations—highlighting their limitations and envisioning next-generation metrics with faithful, natural language explanations to enhance trust and usability in NLP research.

ABSTRACT

Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics for machine translation (for example, COMET or BERTScore) are based on black-box large language models. They often achieve strong correlations with human judgments, but recent research indicates that the lower-quality classical metrics remain dominant, one of the potential reasons being that their decision processes are more transparent. To foster more widespread acceptance of novel high-quality metrics, explainability thus becomes crucial. In this concept paper, we identify key properties as well as key goals of explainable machine translation metrics and provide a comprehensive synthesis of recent techniques, relating them to our established goals and properties. In this context, we also discuss the latest state-of-the-art approaches to explainable metrics based on generative models such as ChatGPT and GPT4. Finally, we contribute a vision of next-generation approaches, including natural language explanations. We hope that our work can help catalyze and guide future research on explainable evaluation metrics and, mediately, also contribute to better and more transparent machine translation systems.

Motivation & Objective

• Address the persistent dominance of low-quality, non-explainable metrics like BLEU in NLP research despite the availability of superior, black-box metrics.
• Identify key challenges in adopting high-quality metrics, particularly the lack of transparency and user trust due to their black-box nature.
• Systematically categorize existing explainability techniques for MT evaluation metrics, focusing on feature importance, perturbation robustness, and LLM-based explanations.
• Propose a vision for next-generation explainable metrics that integrate natural language explanations and improve faithfulness to human judgment.
• Stimulate research toward explainable metrics as essential for trustworthy, auditable, and legally compliant AI in high-stakes applications.

Proposed method

• Survey and categorize existing explainability techniques for MT metrics into three main paradigms: feature attribution (e.g., saliency maps), perturbation-based robustness testing, and LLM-driven explanations.
• Adapt a taxonomy from post-hoc explainability in NLP, organizing methods by model access (white-box vs. black-box) and explanation type (e.g., word-level highlights, error labels).
• Evaluate current approaches using criteria such as faithfulness, human alignment, and computational feasibility, identifying gaps in existing evaluation practices.
• Integrate insights from recent LLM advancements (e.g., GPT-3.5, GPT-4) to enable natural language explanations of metric decisions, moving beyond simple word highlighting.
• Propose a vision for future metrics that combine faithfulness, interpretability, and human-aligned reasoning, using LLMs to generate rationale-like explanations.
• Emphasize the need for rigorous evaluation of explanations themselves, advocating for faithfulness metrics to ensure explanations reflect actual decision processes.

Experimental results

Research questions

• RQ1 What are the key properties and goals of explainable machine translation evaluation metrics, and how do they differ from traditional metrics?
• RQ2 How do current explainability techniques—such as feature attribution and perturbation analysis—perform in explaining scores from state-of-the-art metrics like BERTScore and COMET?
• RQ3 To what extent can large language models (e.g., GPT-4) generate faithful, human-interpretable explanations for MT metric scores?
• RQ4 What are the limitations of existing explainability methods in terms of faithfulness, correspondence between source and target, and generalization across tasks?
• RQ5 What design principles and desiderata should guide the development of next-generation explainable evaluation metrics for MT?

Key findings

• Current high-quality metrics like BERTScore and COMET are underused due to their black-box nature, despite strong correlation with human judgments.
• Feature attribution methods often fail to account for word-level alignment between source and target sentences, leading to misleading error highlights.
• Perturbation-based robustness checks are limited by the need for manually defined perturbation types, which may not generalize across diverse error patterns.
• Recent LLM-based approaches enable fine-grained error labeling and natural language explanations, suggesting a promising shift toward more interpretable evaluation.
• The lack of faithfulness evaluation in current explainability research undermines trust and limits utility in debugging and high-stakes applications.
• Explainability is no longer optional but will become essential for regulatory compliance and trust in AI systems, especially in high-stakes translation tasks.