[Paper Review] ITTR: Unpaired Image-to-Image Translation with Transformers

Wanfeng Zheng, Qiang Li+3 more|arXiv (Cornell University)|Mar 30, 2022

Multimodal Machine Learning Applications12 citations

TL;DR

This paper proposes ITTR, a vision transformer-based architecture for unpaired image-to-image translation that combines a hybrid perception block (HPB) to capture local and global context and dual pruned self-attention (DPSA) to reduce computational cost. ITTR achieves state-of-the-art performance on six benchmarks, outperforming prior methods in both FID and DRN-Score while maintaining efficiency.

ABSTRACT

Unpaired image-to-image translation is to translate an image from a source domain to a target domain without paired training data. By utilizing CNN in extracting local semantics, various techniques have been developed to improve the translation performance. However, CNN-based generators lack the ability to capture long-range dependency to well exploit global semantics. Recently, Vision Transformers have been widely investigated for recognition tasks. Though appealing, it is inappropriate to simply transfer a recognition-based vision transformer to image-to-image translation due to the generation difficulty and the computation limitation. In this paper, we propose an effective and efficient architecture for unpaired Image-to-Image Translation with Transformers (ITTR). It has two main designs: 1) hybrid perception block (HPB) for token mixing from different receptive fields to utilize global semantics; 2) dual pruned self-attention (DPSA) to sharply reduce the computational complexity. Our ITTR outperforms the state-of-the-arts for unpaired image-to-image translation on six benchmark datasets.

Motivation & Objective

• To address the limitation of CNN-based generators in capturing long-range dependencies for unpaired image-to-image translation.
• To overcome the high computational cost and generation instability of recognition-based vision transformers in image translation tasks.
• To design an efficient and effective transformer-based generator that balances global context modeling and computational efficiency.
• To validate the proposed architecture on diverse unpaired image translation benchmarks with strong quantitative and qualitative results.

Proposed method

• The hybrid perception block (HPB) integrates a depth-wise convolution branch for local feature extraction and a self-attention branch for global context modeling, enabling multi-receptive field feature fusion.
• Dual pruned self-attention (DPSA) reduces self-attention complexity by evaluating and pruning low-contribution tokens along row and column directions before computing attention maps.
• Token-wise L2 normalization is applied to query and key matrices in DPSA to stabilize training and prevent collapse, ensuring reliable attention learning.
• The generator architecture is trained end-to-end with cycle-consistency and identity loss, leveraging unpaired data from source and target domains.
• DPSA uses sparse token selection with a learnable contribution score to reduce FLOPs while preserving performance.
• The model is optimized using standard GAN objectives and is evaluated on six standard unpaired translation benchmarks.

Experimental results

Research questions

• RQ1 Can a vision transformer-based generator outperform CNN-based generators in unpaired image-to-image translation by capturing long-range dependencies?
• RQ2 How can the high computational cost of self-attention in vision transformers be reduced without sacrificing translation quality?
• RQ3 What is the impact of combining local perception (CNN) and global attention (transformer) in a single block for image translation?
• RQ4 Does token pruning in self-attention preserve or enhance semantic consistency and image quality in translation tasks?
• RQ5 How does the proposed architecture compare to state-of-the-art methods in terms of FID, DRN-Score, and inference efficiency?

Key findings

• ITTR achieves state-of-the-art performance on six unpaired image-to-image translation benchmarks, outperforming prior methods in both FID and DRN-Score.
• On the Horse→Zebra and Cityscapes datasets, ITTR achieves FID scores of 45.1 and 33.6, respectively, surpassing CUT and LSeSim.
• The ablation study confirms that removing the local perception branch in HPB degrades performance (FID: 45.7 → 48.6), indicating its importance for domain gap reduction.
• DPSA reduces computational complexity significantly: ITTR's generator uses only 45.8 G MACs, less than CUT and LSeSim, while achieving better FID.
• The model collapses without token-wise L2 normalization of Q and K, demonstrating its critical role in training stability.
• Optimal performance is achieved with 8×8 sparse tokens in DPSA; larger token counts (e.g., 16×16) degrade performance due to reduced pruning effect and softmax smoothing.