[Paper Review] Deep Learning Models Delineates Multiple Nuclear Phenotypes in H&E Stained Histology Sections

Mina Khoshdeli, Bahram Parvin|arXiv (Cornell University)|Feb 13, 2018

AI in cancer detection14 references3 citations

TL;DR

This paper proposes a deep learning framework that fuses region-based and boundary-based convolutional neural networks (CNNs) to accurately segment diverse nuclear phenotypes—especially vesicular and overlapping nuclei—in H&E-stained histology slides. The method achieves high-precision segmentation across breast and brain tumor sections, with a 63.2% improvement in separating touching nuclei and F-scores above 0.91 after color decomposition and postprocessing.

ABSTRACT

Nuclear segmentation is an important step for profiling aberrant regions of histology sections. However, segmentation is a complex problem as a result of variations in nuclear geometry (e.g., size, shape), nuclear type (e.g., epithelial, fibroblast), and nuclear phenotypes (e.g., vesicular, aneuploidy). The problem is further complicated as a result of variations in sample preparation. It is shown and validated that fusion of very deep convolutional networks overcomes (i) complexities associated with multiple nuclear phenotypes, and (ii) separation of overlapping nuclei. The fusion relies on integrating of networks that learn region- and boundary-based representations. The system has been validated on a diverse set of nuclear phenotypes that correspond to the breast and brain histology sections.

Motivation & Objective

• Address the challenge of nuclear segmentation in H&E-stained histology due to biological heterogeneity and technical variations in sample preparation.
• Overcome limitations of traditional methods in segmenting complex nuclear phenotypes, particularly vesicular and overlapping nuclei.
• Develop a robust, high-throughput framework capable of processing whole-slide images for large-scale histopathological analysis.
• Integrate region-based and boundary-based representations via a fusion CNN to improve segmentation accuracy and separation of touching nuclei.

Proposed method

• Employ an encoder-decoder U-Net architecture (ENet) for both region-based and boundary-based segmentation networks.
• Train the region-based network to classify pixels as foreground (nuclei) or background using pixel-wise segmentation.
• Train the boundary-based network to detect perceptual boundaries between nuclei, especially in overlapping regions.
• Fuse outputs from region- and boundary-based networks using a third ENet that learns to combine spatial and boundary representations.
• Apply color decomposition to isolate hematoxylin-stained nuclear regions, reducing input complexity and improving training efficiency.
• Use postprocessing with watershed segmentation to resolve remaining overlapping nuclei after fusion network inference.

Experimental results

Research questions

• RQ1 Can a deep learning model effectively segment multiple nuclear phenotypes—including vesicular and aneuploid nuclei—simultaneously in H&E-stained slides?
• RQ2 To what extent can boundary-based representation improve the separation of overlapping or touching nuclei compared to region-based methods alone?
• RQ3 Does the fusion of region- and boundary-based CNNs outperform single-network or traditional segmentation methods like graph cuts in accuracy and robustness?
• RQ4 How does color decomposition affect the performance and generalization of nuclear segmentation models across diverse histology samples?

Key findings

• The fused ENet model achieved an F-score of 0.91 on the test set, significantly outperforming single ENet (F-score 0.85) and multi-reference graph cut (F-score 0.79).
• The model improved separation of overlapping nuclei by 63.2% (62 out of 98 touching nuclei correctly separated), demonstrating strong performance on challenging cases.
• Color decomposition enhanced segmentation performance, with the fused ENet + CD + watershed approach achieving the highest precision (0.94) and recall (0.88).
• The system processed images of size ~360×480 in approximately 60 ms per image on a single GPU, enabling high-throughput analysis of whole-slide images.
• The model successfully segmented difficult phenotypes such as vesicular nuclei and nuclei with high pleomorphism, which are typically poorly handled by conventional methods.
• Postprocessing with watershed segmentation resolved the remaining overlapping nuclei, completing the segmentation pipeline with high fidelity.