[Paper Review] Learning Mutually Local-global U-nets For High-resolution Retinal Lesion Segmentation in Fundus Images

Zizheng Yan, Xiaoguang Han+4 more|arXiv (Cornell University)|Jan 18, 2019

Retinal Imaging and Analysis12 references19 citations

TL;DR

This paper proposes a mutually enhanced local-global U-Net framework that jointly optimizes a global-downsampled U-Net and a patch-based U-Net to improve high-resolution retinal lesion segmentation. By fusing feature maps from the global decoder into the local decoder and jointly training both branches, the model achieves superior performance on small, scattered lesions like microaneurysms and hard exudates, outperforming both standalone global and local networks.

ABSTRACT

Diabetic retinopathy is the most important complication of diabetes. Early diagnosis of retinal lesions helps to avoid visual loss or blindness. Due to high-resolution and small-size lesion regions, applying existing methods, such as U-Nets, to perform segmentation on fundus photography is very challenging. Although downsampling the input images could simplify the problem, it loses detailed information. Conducting patch-level analysis helps reaching fine-scale segmentation yet usually leads to misunderstanding as the lack of context information. In this paper, we propose an efficient network that combines them together, not only being aware of local details but also taking fully use of the context perceptions. This is implemented by integrating the decoder parts of a global-level U-net and a patch-level one. The two streams are jointly optimized, ensuring that they are enhanced mutually. Experimental results demonstrate our new framework significantly outperforms existing patch-based and global-based methods, especially when the lesion regions are scattered and small-scaled.

Motivation & Objective

• To address the challenge of segmenting small, scattered retinal lesions in high-resolution fundus images where global methods lose detail and local methods lack context.
• To overcome the limitations of existing patch-based and global-based U-Net approaches by integrating local fine-grained features with global contextual awareness.
• To design a unified, end-to-end trainable framework that enables mutual enhancement between local and global segmentation branches.
• To improve segmentation accuracy for microaneurysms and hard exudates—lesions that are small and scattered—where current methods struggle due to resolution and context trade-offs.

Proposed method

• The framework consists of two U-Net branches: a GlobalNet that processes downsampled images to produce coarse segmentation maps, and a LocalNet that processes cropped patches for fine-grained segmentation.
• Feature maps from the GlobalNet decoder are cropped and concatenated into the corresponding layers of the LocalNet decoder to inject global context into local predictions.
• The two branches are jointly optimized via a shared loss function, enabling mutual enhancement: local losses improve global feature learning, and global context reduces local ambiguity.
• A three-stage training strategy is employed: pre-train GlobalNet and LocalNet separately, then fine-tune the fusion module, and finally end-to-end fine-tune the entire network.
• The final segmentation is obtained by stitching predictions from all patches, with inference on each 256×256 patch taking 10–30 ms.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework effectively combine local detail sensitivity and global context awareness for high-resolution retinal lesion segmentation?
• RQ2 Does joint optimization of local and global U-Net branches lead to improved performance on small, scattered lesions compared to isolated local or global models?
• RQ3 How does the fusion of global context into local segmentation reduce misclassification and inconsistency across patch boundaries?
• RQ4 In what lesion types (e.g., microaneurysms vs. hemorrhages) does the proposed method show the greatest performance gain?

Key findings

• The proposed fused model achieves an AUPR of 0.889 for hard exudates (EX), significantly outperforming GlobalNet (0.849) and LocalNet (0.845).
• For microaneurysms (MA), the fused model reaches an AUPR of 0.525, surpassing GlobalNet (0.484) and LocalNet (0.433).
• The model performs best on scattered, small lesions: it improves performance on EX and MA, where global methods lose detail and local methods lack context.
• For larger, more compact lesions like hemorrhages (HE) and soft exudates (SE), GlobalNet still outperforms the fused model, indicating the method is most effective for small, scattered targets.
• The model achieves state-of-the-art performance on the 2018 ISBI grand challenge leaderboard, exceeding all reported results using U-Net as the backbone.
• The three-stage training strategy—pre-training, fusion module fine-tuning, and end-to-end fine-tuning—ensures stable convergence and effective feature transfer between branches.