[Paper Review] Improving CAT Tools in the Translation Workflow: New Approaches and Evaluation

Mihaela Vela, Santanu Pal+3 more|arXiv (Cornell University)|Aug 16, 2019

Natural Language Processing Techniques7 citations

TL;DR

This paper introduces CATaLog Online, a web-based CAT tool that enhances translation post-editing through a novel color-coding scheme for TM suggestions and improved similarity-based retrieval using TER, Needleman-Wunsch, and Lucene scores. User studies show translators prefer MT output and find the color-coded interface significantly improves decision-making and editing efficiency, though usability is limited by missing features like spell-check and keyboard shortcuts.

ABSTRACT

This paper describes strategies to improve an existing web-based computer-aided translation (CAT) tool entitled CATaLog Online. CATaLog Online provides a post-editing environment with simple yet helpful project management tools. It offers translation suggestions from translation memories (TM), machine translation (MT), and automatic post-editing (APE) and records detailed logs of post-editing activities. To test the new approaches proposed in this paper, we carried out a user study on an English--German translation task using CATaLog Online. User feedback revealed that the users preferred using CATaLog Online over existing CAT tools in some respects, especially by selecting the output of the MT system and taking advantage of the color scheme for TM suggestions.

Motivation & Objective

• To improve translation post-editing productivity by enhancing CAT tool interfaces and TM retrieval strategies.
• To evaluate the impact of visual design—specifically color coding—on translator decision-making and editing efficiency.
• To compare user preferences between MT, TM, and APE outputs in a real-world post-editing setting.
• To identify usability limitations in current CAT tools and propose improvements based on translator feedback.
• To collect detailed post-editing logs for incremental MT/APE and translation process research.

Proposed method

• CATaLog Online integrates translation suggestions from TM, MT, and APE engines within a single web-based interface.
• A novel intra-segment color-coding scheme highlights matching and irrelevant fragments in TM suggestions to guide post-editing.
• Similarity matching combines TER, Needleman-Wunsch alignment, and Lucene retrieval scores to re-rank TM matches for relevance.
• The system logs all post-editing activities in structured XML format for downstream analysis and incremental learning.
• A user study was conducted with professional translators on an English–German translation task to evaluate usability and preference.
• Feedback was collected on interface design, feature utility, and perceived cognitive load, with focus on time-effort correlation and edit patterns.

Experimental results

Research questions

• RQ1 How does color-coded presentation of TM suggestions affect translator decision-making and editing efficiency?
• RQ2 What is the relative preference of translators for MT, TM, or APE output in a post-editing workflow?
• RQ3 To what extent does the number of edits correlate with post-editing time, and how does this vary by translator?
• RQ4 How do usability limitations such as missing keyboard shortcuts or spell-check affect user experience in CAT tools?
• RQ5 Can detailed logging of post-editing activities support incremental MT and APE system development?

Key findings

• Translators showed a clear preference for using MT output over TM suggestions, even when TM matches were semantically relevant.
• The color-coding scheme for TM fragments significantly aided translators in identifying which parts to edit, improving decision-making efficiency.
• There was a low correlation between the number of edits and post-editing time, indicating that editing time is a subjective, translator-dependent measure.
• The most frequent edit types were substitutions, followed by insertions, deletions, and shifts, reflecting common post-editing patterns.
• Users reported that the interface layout and suggestion arrangement were helpful, but criticized the lack of spell-checker, keyboard shortcuts, and concordancer functionality.
• The system's detailed logging of post-editing activities in XML format provides a valuable resource for future research in incremental MT and APE training.