[Paper Review] MSFNet-CPD: Multi-Scale Cross-Modal Fusion Network for Crop Pest Detection

Jiaqi Zhang, Zhuodong Liu+1 more|ArXiv.org|May 5, 2025

Smart Agriculture and AI3 citations

TL;DR

MSFNet-CPD introduces a multi-scale cross-modal fusion network that combines image and text features, with super-resolution and data augmentation, achieving superior pest detection on IP102-derived multi-modal datasets.

ABSTRACT

Accurate identification of agricultural pests is essential for crop protection but remains challenging due to the large intra-class variance and fine-grained differences among pest species. While deep learning has advanced pest detection, most existing approaches rely solely on low-level visual features and lack effective multi-modal integration, leading to limited accuracy and poor interpretability. Moreover, the scarcity of high-quality multi-modal agricultural datasets further restricts progress in this field. To address these issues, we construct two novel multi-modal benchmarks-CTIP102 and STIP102-based on the widely-used IP102 dataset, and introduce a Multi-scale Cross-Modal Fusion Network (MSFNet-CPD) for robust pest detection. Our approach enhances visual quality via a super-resolution reconstruction module, and feeds both the original and reconstructed images into the network to improve clarity and detection performance. To better exploit semantic cues, we propose an Image-Text Fusion (ITF) module for joint modeling of visual and textual features, and an Image-Text Converter (ITC) that reconstructs fine-grained details across multiple scales to handle challenging backgrounds. Furthermore, we introduce an Arbitrary Combination Image Enhancement (ACIE) strategy to generate a more complex and diverse pest detection dataset, MTIP102, improving the model's generalization to real-world scenarios. Extensive experiments demonstrate that MSFNet-CPD consistently outperforms state-of-the-art methods on multiple pest detection benchmarks. All code and datasets will be made publicly available at: https://github.com/Healer-ML/MSFNet-CPD.

Motivation & Objective

• Motivate robust crop pest detection amidst high intra-class variance and complex backgrounds.
• Leverage multi-modal cues by fusing visual and textual pest descriptions.
• Enhance image quality via super-resolution to preserve high-frequency pest features.
• Create and evaluate multi-modal benchmarks CTIP102 and STIP102 from IP102, plus MTIP102 with ACIE augmentation.
• Demonstrate improved detection accuracy over unimodal baselines across multiple datasets.

Proposed method

• LS RGAN for super-resolution to generate high-quality images while keeping original images as supplementary features.
• TIC (Text-Image Converter) to extract multi-scale visual features and align them with text for ITF input.
• ITF (Image-Text Fusion) Transformer-based module to fuse multi-scale visual features with text features.
• ITC (Image-Text Converter) to reconstruct multi-scale features back to image-like representations for downstream PTI.
• PTI (Pest Target Identification) neck and head based on YOLOv4-style architecture to predict pest bounding boxes across scales.
• ACIE (Arbitrary Combination Image Enhancement) data augmentation to create MTIP102 with multiple pests and varied scales.

Figure 1 : MSFNet-CPD Model Architecture (A. LSRGAN for super-resolution of low-quality images, B. Picture to ITF Transformer (TIC), C. Multi-scale Cross-modal Fusion ITF Module, D. ITF to Neck Network Converter (ITC), and E. Pest Target Identification (PTI). Numbers represent categories in Original

Experimental results

Research questions

• RQ1 Can multi-modal learning with image and text descriptions improve pest detection accuracy over unimodal methods on IP102-derived datasets?
• RQ2 Do high-frequency preserved via super-resolution and cross-scale fusion better support detection in cluttered agricultural environments?
• RQ3 How does text description complexity influence detection performance in multi-modal pest identification?
• RQ4 What is the impact of data augmentation (ACIE) in generating realistic, multi-target pest images on model generalization?

Key findings

• MSFNet-CPD achieves mAP of 46.06% and mAP 50 of 92.18% when using both image and text inputs.
• Removing text reduces mAP by 10.42% and removing LSRGAN reduces mAP by 11.24%, showing the importance of text and super-resolution.
• CTIP102 and STIP102 outperform the unimodal IP102 baseline in detection performance.
• Complex text descriptions yield higher Precision, mAP, and mAP 50 than simple descriptions (improvement in Precision 9.72%, mAP 1.08%, mAP 50 3.96%).
• YOLOv9 baseline mAP is 42.32% with mAP50 81.10% and mAP75 39.86%, while MSFNet-CPD surpasses these metrics on multi-modal datasets.

Figure 2 : ITF Specific Process. ( $B_{i}$ , $C_{i}$ , $V_{i}$ stand for different scale image features and $T$ stands for text features).