[Paper Review] Learning from Multiple Datasets with Heterogeneous and Partial Labels for Universal Lesion Detection in CT

Ke Yan, Jinzheng Cai+8 more|arXiv (Cornell University)|Sep 5, 2020

Radiomics and Machine Learning in Medical Imaging67 references9 citations

TL;DR

This paper proposes Lesion ENSemble (LENS), a multi-task learning framework that unifies heterogeneous and partially labeled CT datasets for universal lesion detection. By fusing proposals across multiple dataset-specific detectors and mining missing annotations using clinical priors and cross-dataset knowledge transfer, LENS achieves a 49% relative improvement in average sensitivity over state-of-the-art methods on DeepLesion.

ABSTRACT

Large-scale datasets with high-quality labels are desired for training accurate deep learning models. However, due to the annotation cost, datasets in medical imaging are often either partially-labeled or small. For example, DeepLesion is such a large-scale CT image dataset with lesions of various types, but it also has many unlabeled lesions (missing annotations). When training a lesion detector on a partially-labeled dataset, the missing annotations will generate incorrect negative signals and degrade the performance. Besides DeepLesion, there are several small single-type datasets, such as LUNA for lung nodules and LiTS for liver tumors. These datasets have heterogeneous label scopes, i.e., different lesion types are labeled in different datasets with other types ignored. In this work, we aim to develop a universal lesion detection algorithm to detect a variety of lesions. The problem of heterogeneous and partial labels is tackled. First, we build a simple yet effective lesion detection framework named Lesion ENSemble (LENS). LENS can efficiently learn from multiple heterogeneous lesion datasets in a multi-task fashion and leverage their synergy by proposal fusion. Next, we propose strategies to mine missing annotations from partially-labeled datasets by exploiting clinical prior knowledge and cross-dataset knowledge transfer. Finally, we train our framework on four public lesion datasets and evaluate it on 800 manually-labeled sub-volumes in DeepLesion. Our method brings a relative improvement of 49% compared to the current state-of-the-art approach in the metric of average sensitivity. We have publicly released our manual 3D annotations of DeepLesion in https://github.com/viggin/DeepLesion_manual_test_set.

Motivation & Objective

• To address the challenge of training universal lesion detectors from multiple medical imaging datasets with heterogeneous and partial labels.
• To improve lesion detection performance by leveraging synergies across diverse datasets with varying label scopes and annotation quality.
• To develop a method that can detect a wide range of lesion types across different organs in CT scans, mimicking radiologist workflows.
• To mitigate performance degradation caused by missing annotations in large-scale datasets like DeepLesion.

Proposed method

• LENS employs a multi-task learning framework that shares features across multiple dataset-specific lesion detectors while fusing region proposals from each expert to enhance detection sensitivity.
• It uses a proposal fusion strategy that aggregates detections from different dataset experts to improve recall across lesion types.
• Missing annotations in partially-labeled datasets are mined using clinical prior knowledge, such as lesion size, location, and anatomical plausibility.
• Cross-dataset knowledge transfer is applied to transfer detection knowledge from well-annotated datasets (e.g., LUNA, LiTS) to under-annotated ones (e.g., DeepLesion) to improve feature learning.
• A false positive reduction classifier is used to suppress erroneous detections from individual experts, improving precision.
• The framework is trained on four public datasets (DeepLesion, LUNA, LiTS, and others) and evaluated on a manually curated test set of 800 sub-volumes from DeepLesion.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework effectively learn from multiple heterogeneous lesion datasets with varying label scopes and annotation completeness?
• RQ2 How can missing annotations in partially-labeled datasets be effectively recovered without manual re-annotation?
• RQ3 Can multi-dataset joint learning improve detection performance for both well- and poorly-annotated lesion types?
• RQ4 To what extent does proposal fusion across multiple dataset-specific experts enhance sensitivity in universal lesion detection?
• RQ5 Can clinical prior knowledge and cross-dataset knowledge transfer improve detection performance in data-scarce scenarios?

Key findings

• LENS achieved a 49% relative improvement in average sensitivity over the state-of-the-art method on the DeepLesion test set, demonstrating significant performance gains.
• Joint training on multiple datasets consistently improved detection performance across all single-type datasets, especially with limited training data.
• The model achieved an average sensitivity of 0.898 on the LUNA benchmark under official evaluation metrics, ranking 6th among 62 teams without model ensembling or specialized post-processing.
• The integration of multiple dataset experts via proposal fusion significantly increased recall, particularly for rare or subtle lesions missed by single-expert models.
• The use of clinical priors and cross-dataset knowledge transfer enabled effective mining of missing annotations, reducing the negative impact of partial labeling.
• The framework demonstrated robustness to distribution shifts across datasets, with improved generalization across diverse lesion types and anatomical locations.