[Paper Review] A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges

Huangjun Shen, Liangying Shao+4 more|arXiv (Cornell University)|May 21, 2024

Natural Language Processing Techniques4 citations

TL;DR

This survey provides a comprehensive analysis of multi-modal machine translation (MMT), covering 99 prior works across tasks, models, datasets, and evaluation metrics. It identifies key techniques like dual attention mechanisms and contrastive learning, highlights performance gains from visual context, and outlines future directions in LLM integration, better evaluation, and scalable datasets.

ABSTRACT

In recent years, multi-modal machine translation has attracted significant interest in both academia and industry due to its superior performance. It takes both textual and visual modalities as inputs, leveraging visual context to tackle the ambiguities in source texts. In this paper, we begin by offering an exhaustive overview of 99 prior works, comprehensively summarizing representative studies from the perspectives of dominant models, datasets, and evaluation metrics. Afterwards, we analyze the impact of various factors on model performance and finally discuss the possible research directions for this task in the future. Over time, multi-modal machine translation has developed more types to meet diverse needs. Unlike previous surveys confined to the early stage of multi-modal machine translation, our survey thoroughly concludes these emerging types from different aspects, so as to provide researchers with a better understanding of its current state.

Motivation & Objective

• To provide a systematic and up-to-date review of multi-modal machine translation (MMT) research, including emerging types beyond scene-image MMT.
• To analyze the impact of various model architectures, training strategies, and visual encoding techniques on MMT performance.
• To identify gaps in current evaluation practices, particularly the overreliance on text-only metrics like BLEU and METEOR.
• To propose future research directions, including integration with large language models (LLMs), improved automatic evaluation, and scalable, multi-domain datasets.

Proposed method

• The survey conducts a comprehensive literature review of 99 representative papers in MMT, categorized by task type, model design, training strategy, and analysis.
• It classifies MMT into scene-image MMT and emerging types such as e-commerce product MT, video-guided MT, and multi-modal simultaneous translation.
• Key model design techniques include dual attention mechanisms, image-as-supplement, text-to-image generation, and retrieval-based methods.
• It evaluates performance using standard metrics like BLEU and METEOR, with detailed comparison of results across datasets and model variants.
• The study analyzes the effectiveness of image encoders (e.g., ResNet, Vision Transformers) and training techniques (e.g., multi-task learning, contrastive learning, unsupervised learning).
• It discusses the limitations of current evaluation, advocating for vision-grounded metrics and LLM-based evaluation to better assess semantic alignment with visual inputs.

Experimental results

Research questions

• RQ1 How do different model architectures—such as dual attention mechanisms or image-supplement strategies—affect translation quality in MMT?
• RQ2 What is the relative performance of various image encoding methods (e.g., ResNet, Vision Transformers) and training techniques (e.g., contrastive learning, pre-training) in MMT?
• RQ3 Why are current automatic evaluation metrics like BLEU and METEOR insufficient for MMT, and what improvements are needed?
• RQ4 How can large language models (LLMs) be better integrated with visual modalities to enhance MMT performance?
• RQ5 What are the key challenges in scaling MMT to real-world applications, and how can high-quality, multi-domain datasets be constructed?

Key findings

• The use of visual context significantly improves translation accuracy, especially for ambiguous words—e.g., 'course' meaning 'route' in a maritime context is correctly interpreted with image support.
• Model architectures using dual attention mechanisms (e.g., DA+HAN) achieve BLEU scores of up to 35.9 on the VaTex dataset, outperforming simpler fusion methods.
• Video-guided MMT models using Faster R-CNN and ResNet with reinforcement learning reach BLEU scores of 59.1 on the English-to-French translation task.
• Multi-modal simultaneous translation (SiMT) models with wait-k decoding strategies achieve BLEU scores of up to 56.2 on the Test2016 set, showing improved latency-performance trade-offs.
• Contrastive learning and multi-task training consistently improve performance across multiple datasets, with gains observed in both scene-image and video-guided MMT.
• Existing evaluation metrics fail to capture visual-textual semantic alignment, highlighting the urgent need for vision-aware metrics and LLM-based evaluation frameworks.