[Paper Review] Deep-learning-based identification of odontogenic keratocysts in hematoxylin- and eosin-stained jaw cyst specimens

Kei Sakamoto, Keiichi Morita+2 more|arXiv (Cornell University)|Jan 12, 2019

Oral and Maxillofacial Pathology20 references4 citations

TL;DR

This study develops a deep learning system using VGG16 fine-tuned on H&E-stained microscopy images to automatically identify odontogenic keratocysts by detecting the diagnostic basal cell palisading pattern. The model achieved an AUC of 0.997 on a test set, demonstrating high accuracy for automated keratocyst diagnosis despite challenges in handling subtle histological variations.

ABSTRACT

The aim of this study was to develop a digital histopathology system for identifying odontogenic keratocysts in hematoxylin- and eosin-stained tissue specimens of jaw cysts. Approximately 5000 microscopy images with 400$ imes$ magnification were obtained from 199 odontogenic keratocysts, 208 dentigerous cysts, and 55 radicular cysts. A proportion of these images were used to make training patches, which were annotated as belonging to one of the following three classes: keratocysts, non-keratocysts, and stroma. The patches for the cysts contained the complete lining epithelium, with the cyst cavity being present on the upper side. The convolutional neural network (CNN) VGG16 was finetuned to this dataset. The trained CNN could recognize the basal cell palisading pattern, which is the definitive criterion for diagnosing keratocysts. Some of the remaining images were scanned and analyzed by the trained CNN, whose output was then used to train another CNN for binary classification (keratocyst or not). The area under the receiver operating characteristics curve for the entire algorithm was 0.997 for the test dataset. Thus, the proposed patch classification strategy is usable for automated keratocyst diagnosis. However, further optimization must be performed to make it suitable for practical use.

Motivation & Objective

• To develop an automated digital histopathology system for identifying odontogenic keratocysts in routine H&E-stained tissue sections.
• To address the diagnostic challenge posed by the subjectivity and variability in keratocyst identification, especially among less experienced pathologists.
• To evaluate whether deep learning can autonomously learn and detect the definitive histologic feature—basal cell palisading—critical for keratocyst diagnosis.
• To explore the feasibility of using patch-based convolutional neural networks (CNNs) for classifying rare oral cysts with limited training data.
• To assess whether the model's decision-making process aligns with pathologic reasoning by analyzing feature map activations.

Proposed method

• A total of 5,000 microscopy images at 400× magnification were collected from 199 odontogenic keratocysts, 208 dentigerous cysts, and 55 radicular cysts.
• Training patches were extracted and annotated into three classes: keratocysts, non-keratocysts, and stroma, with complete epithelial lining and cyst cavity preserved.
• The VGG16 CNN was fine-tuned on these annotated patches to learn discriminative features, particularly the palisading pattern.
• A second CNN was trained on whole-slide image outputs to perform binary classification (keratocyst vs. non-keratocyst).
• Model performance was evaluated using the area under the ROC curve (AUC) on a held-out test set.
• Interpretability was assessed by analyzing activation patterns in intermediate convolutional layers to identify which features the network prioritized.

Experimental results

Research questions

• RQ1 Can a deep learning model accurately identify odontogenic keratocysts in routine H&E-stained histopathology slides?
• RQ2 Does the model learn to detect the definitive histologic feature—basal cell palisading—used by pathologists for keratocyst diagnosis?
• RQ3 To what extent does the model’s internal feature learning align with expert diagnostic reasoning?
• RQ4 Can patch-based CNNs achieve high diagnostic accuracy for rare oral cysts despite limited training data?
• RQ5 Are there histologic features overlooked by human pathologists that the model discovers through end-to-end learning?

Key findings

• The deep learning model achieved an area under the ROC curve (AUC) of 0.997 on the test dataset, indicating near-perfect diagnostic performance.
• The model successfully learned to detect the basal cell palisading pattern, the definitive histologic criterion for odontogenic keratocysts, through activation analysis of intermediate layers.
• Filters in the final convolutional layer responded strongly to aligned elliptical nuclei in the basal layer, confirming the model’s focus on the key diagnostic feature.
• The model discovered a previously overlooked histologic feature: enhanced signal in spinous layer filters, likely reflecting high protein synthesis activity due to keratinization, which is not emphasized in standard textbooks.
• Despite high overall accuracy, the model struggled to distinguish keratocysts with poor palisading or non-keratocysts with pseudo-palisading patterns, indicating limitations in handling ambiguous cases.
• The activation patterns suggest the model’s reasoning is biologically plausible, as it prioritizes features analogous to those pathologists use, though it may miss subtle architectural cues used by experts.