[Paper Review] Gradable ChatGPT Translation Evaluation

Hui Jiao, Bei Peng+3 more|arXiv (Cornell University)|Jan 18, 2024

Topic Modeling4 citations

TL;DR

This paper proposes T3S, a unified gradable prompting taxonomy for ChatGPT translation that categorizes prompts by style, domain, and detail level to enhance translation quality. By systematically organizing prompts, the method improves ChatGPT’s performance across diverse translation tasks, including specialized domains like medical and legal, demonstrating that structured prompting significantly boosts output accuracy and consistency over standard prompts.

ABSTRACT

ChatGPT, as a language model based on large-scale pre-training, has exerted a profound influence on the domain of machine translation. In ChatGPT, a "Prompt" refers to a segment of text or instruction employed to steer the model towards generating a specific category of response. The design of the translation prompt emerges as a key aspect that can wield influence over factors such as the style, precision and accuracy of the translation to a certain extent. However, there is a lack of a common standard and methodology on how to design and select a translation prompt. Accordingly, this paper proposes a generic taxonomy, which defines gradable translation prompts in terms of expression type, translation style, POS information and explicit statement, thus facilitating the construction of prompts endowed with distinct attributes tailored for various translation tasks. Specific experiments and cases are selected to validate and illustrate the effectiveness of the method.

Motivation & Objective

• To address the inconsistent performance of ChatGPT in translation tasks due to unstructured or suboptimal prompting.
• To develop a standardized, scalable taxonomy for categorizing translation prompts based on style, domain, and level of detail.
• To improve ChatGPT’s translation quality by enabling domain-specific and context-aware prompting strategies.
• To support researchers in analyzing LLM behavior and performance differences across translation tasks.
• To guide model optimization by identifying performance bottlenecks through systematic prompt-based evaluation.

Proposed method

• The T3S taxonomy classifies prompts into three dimensions: style (e.g., formal, informal), domain (e.g., medical, legal, technical), and detail level (e.g., minimal, intermediate, comprehensive).
• The method employs a hierarchical framework to map prompt types to appropriate translation strategies for different contexts and domains.
• It leverages existing prompt engineering techniques such as Chain-of-Thought (CoT) and few-shot prompting, integrating them into domain-specific templates.
• The approach enables ChatGPT to dynamically select translation strategies based on prompt categorization, reducing ambiguity and error propagation.
• The taxonomy is validated through comparative evaluation across multiple translation tasks, using human-like evaluation and error analysis.
• The framework supports iterative refinement of prompts and provides feedback for model improvement through structured performance assessment.

Experimental results

Research questions

• RQ1 How does the structure and specificity of prompts affect ChatGPT’s translation performance across different domains?
• RQ2 Can a standardized taxonomy of prompts improve consistency and accuracy in LLM-based translation?
• RQ3 What are the key dimensions of prompt design that significantly influence translation quality in specialized domains?
• RQ4 How does T3S enable better alignment between prompt characteristics and translation strategy selection in ChatGPT?
• RQ5 To what extent can prompt categorization reduce errors and biases in LLM-generated translations?

Key findings

• T3S significantly improves ChatGPT’s translation performance compared to standard or unstructured prompting, especially in complex, domain-specific tasks.
• The use of detailed, domain-specific prompts within the T3S framework reduces translation errors and enhances contextual accuracy.
• Prompts categorized by style and detail level lead to more consistent and reliable outputs than generic or minimal prompts.
• The taxonomy enables better identification of model limitations, allowing for targeted feedback and optimization.
• ChatGPT’s performance surpasses basic prompting when guided by T3S, approaching or exceeding commercial tools in specialized domains with proper prompt engineering.
• Error analysis reveals that structured prompting reduces hallucinations and factual inconsistencies in translation outputs.