[Paper Review] PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion

Yu Fu, Tianyang Xu+2 more|arXiv (Cornell University)|Jul 29, 2021

Advanced Image Fusion TechniquesEngineering46 references44 citations

TL;DR

The paper introduces a Pyramid Patch Transformer (PPT) that combines local Patch Transformer and global Pyramid Transformer to extract multi-scale, multi-level features for low-level vision tasks, and applies it to image fusion with competitive results.

ABSTRACT

The Transformer architecture has witnessed a rapid development in recent years, outperforming the CNN architectures in many computer vision tasks, as exemplified by the Vision Transformers (ViT) for image classification. However, existing visual transformer models aim to extract semantic information for high-level tasks, such as classification and detection.These methods ignore the importance of the spatial resolution of the input image, thus sacrificing the local correlation information of neighboring pixels. In this paper, we propose a Patch Pyramid Transformer(PPT) to effectively address the above issues.Specifically, we first design a Patch Transformer to transform the image into a sequence of patches, where transformer encoding is performed for each patch to extract local representations. In addition, we construct a Pyramid Transformer to effectively extract the non-local information from the entire image. After obtaining a set of multi-scale, multi-dimensional, and multi-angle features of the original image, we design the image reconstruction network to ensure that the features can be reconstructed into the original input. To validate the effectiveness, we apply the proposed Patch Pyramid Transformer to image fusion tasks. The experimental results demonstrate its superior performance, compared to the state-of-the-art fusion approaches, achieving the best results on several evaluation indicators. Thanks to the underlying representational capacity of the PPT network, it can directly be applied to different image fusion tasks without redesigning or retraining the network.

Motivation & Objective

• Motivate and address the limitation of pure global transformers in low-level vision tasks by preserving local pixel-level information.
• Develop a Patch Transformer to model pixel-level correlations within patches.
• Construct a Pyramid Transformer to capture global, multi-scale relationships between patches.
• Integrate Patch and Pyramid Transformers into an auto-encoder for robust image reconstruction.
• Demonstrate PPT’s effectiveness across various image fusion tasks without task-specific redesigns.

Proposed method

• Introduce Patch Transformer that processes each patch with transformer encoding to extract local representations of all pixels within the patch.
• Build a Pyramid Transformer by down-sampling and applying Patch Transformer at multiple scales, then up-sampling and concatenating features to form multi-scale representations.
• Assemble an auto-encoder architecture where the Pyramid and Patch Transformers form the encoder and an MLP-based decoder reconstructs the image (loss = MSE).
• Apply the PPT encoder to multi-source images (e.g., infrared and visible) in a Siamese setup and fuse features (F_fused) via channel-wise fusion strategies.
• Use three fusion strategies (average, maximum, Softmax) to adaptively combine features prior to decoding into a fused image.

Experimental results

Research questions

• RQ1 Can a Transformer-based model be effectively used for low-level vision tasks by preserving local pixel information within patches?
• RQ2 Does a multi-scale Pyramid Transformer improve global context modeling without sacrificing local texture details?
• RQ3 How well does the PPT-based feature extractor perform in multi-source image fusion across infrared/visible, multi-focus, and medical datasets compared to state-of-the-art methods?
• RQ4 Is the proposed architecture general enough to apply to different image fusion tasks without network redesign?
• RQ5 Which fusion strategy (average, max, Softmax) yields best quantitative fusion metrics across datasets?

Key findings

• PPT can extract both local texture and global contextual features, enabling effective low-level vision representations.
• The Pyramid Patch Transformer yields multi-scale features that improve fusion quality while remaining CNN-free.
• On infrared/visible fusion tasks, PPT Fusion ranks in the top-2 across multiple indicators on datasets like TNO and RoadScene.
• For multi-focus and other fusion tasks, the method achieves competitive or superior scores compared to a broad set of state-of-the-art methods (as reported in the paper's comparative analysis).
• The approach requires less computational resources than traditional large-scale transformers and can operate with modest hardware while using COCO/Imagenet pretraining.
• The authors report that PPT Fusion achieves best or near-best performance on several quantitative metrics (SCD, SSIM, CC, FMI_pixel, etc.) and demonstrates favorable qualitative fusion results.