[Paper Review] From Patch to Image Segmentation using Fully Convolutional Networks -- Application to Retinal Images

Taibou Birgui Sekou, Moncef Hidane+2 more|arXiv (Cornell University)|Apr 8, 2019

Retinal Imaging and Analysis39 references22 citations

TL;DR

This paper proposes a novel training framework that combines patch-based pre-training with image-level fine-tuning using fully convolutional networks (FCNs) to achieve state-of-the-art retinal image segmentation on small datasets. By first training a network on patches and then transferring knowledge to an image-level network, the method enables arbitrary network architectures and achieves high performance with minimal data, validated across four public retinal databases using lightweight, standard, and densely connected networks.

ABSTRACT

Deep learning based models, generally, require a large number of samples for appropriate training, a requirement that is difficult to satisfy in the medical field. This issue can usually be avoided with a proper initialization of the weights. On the task of medical image segmentation in general, two techniques are oftentimes employed to tackle the training of a deep network $f_T$. The first one consists in reusing some weights of a network $f_S$ pre-trained on a large scale database ($e.g.$ ImageNet). This procedure, also known as $transfer$ $learning$, happens to reduce the flexibility when it comes to new network design since $f_T$ is constrained to match some parts of $f_S$. The second commonly used technique consists in working on image patches to benefit from the large number of available patches. This paper brings together these two techniques and propose to train $arbitrarily$ $designed$ $networks$ that segment an image in one forward pass, with a focus on relatively small databases. An experimental work have been carried out on the tasks of retinal blood vessel segmentation and the optic disc one, using four publicly available databases. Furthermore, three types of network are considered, going from a very light weighted network to a densely connected one. The final results show the efficiency of the proposed framework along with state of the art results on all the databases.

Motivation & Objective

• To address the challenge of limited annotated medical image data in training deep networks for retinal image segmentation.
• To overcome the limitations of standard transfer learning, which constrains network design due to weight reuse from pre-trained models like ImageNet.
• To develop a flexible training framework that allows arbitrary network architecture design while maintaining high performance on small medical datasets.
• To improve segmentation accuracy and robustness by leveraging contextual information through image-level inference after patch-based pre-training.
• To enable real-time, end-to-end segmentation by training networks that process full images in a single forward pass.

Proposed method

• The framework first trains a fully convolutional network (FCN) on small image patches extracted from retinal images, using a patch-to-patch classification approach.
• After pre-training on patches, the network's learned features are transferred to a new, larger FCN architecture designed for full-image inference.
• The image-level network is initialized with the pre-trained patch network's encoder weights, enabling effective feature transfer without architectural constraints.
• The image-level network is then fine-tuned end-to-end on full-sized images using standard supervised learning with cross-entropy or dice loss.
• The method supports various network architectures—from lightweight to densely connected—by decoupling pre-training from final network design.
• A resizing step is applied during training to reduce memory usage and improve numerical stability, with minimal impact on performance.

Experimental results

Research questions

• RQ1 Can a patch-based pre-training strategy combined with image-level fine-tuning achieve state-of-the-art performance in retinal image segmentation with limited training data?
• RQ2 Does the proposed framework allow for the use of arbitrarily designed FCN architectures without being constrained by pre-trained ImageNet weights?
• RQ3 How does the performance of the patch-to-image transfer framework compare to training from scratch or standard transfer learning on small retinal image databases?
• RQ4 To what extent does the framework preserve contextual information and reduce boundary artifacts in segmentation maps compared to patch-based inference?
• RQ5 Can the framework be effectively applied to diverse retinal image tasks such as blood vessel and optic disc segmentation across multiple public datasets?

Key findings

• The proposed framework achieves state-of-the-art performance on four public retinal image databases: DRIVE, STARE, CHASE_DB1, and IDRiD.
• The method reduces segmentation inference time significantly compared to patch-based aggregation, with image-level inference taking only 0.24 seconds on average using the Dense network on DRIVE.
• Fine-tuning after frozen transfer improves segmentation quality, as evidenced by reduced false positives and stronger boundary predictions in qualitative results.
• The framework maintains high performance even with image resizing for memory efficiency, demonstrating robustness to preprocessing steps.
• The approach outperforms training from scratch and standard transfer learning, especially in low-data regimes, due to effective feature transfer from patch-level learning.
• The source code is publicly available at https://github.com/Taib/patch2image, enabling reproducibility and further research.