[Paper Review] BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once

Theodore Zhao, 裕二 池谷+13 more|arXiv (Cornell University)|May 21, 2024

AI in cancer detection4 citations

TL;DR

BiomedParse is a biomedical foundation model that jointly performs segmentation, detection, and recognition across 82 object types and 9 imaging modalities using a unified text prompt, eliminating the need for user-provided bounding boxes. It achieves state-of-the-art performance by leveraging GPT-4 to align noisy, unstructured dataset descriptions with biomedical ontologies, creating a large-scale dataset of over 6 million image-mask-description triples for pretraining.

ABSTRACT

Biomedical image analysis is fundamental for biomedical discovery in cell biology, pathology, radiology, and many other biomedical domains. Holistic image analysis comprises interdependent subtasks such as segmentation, detection, and recognition of relevant objects. Here, we propose BiomedParse, a biomedical foundation model for imaging parsing that can jointly conduct segmentation, detection, and recognition for 82 object types across 9 imaging modalities. Through joint learning, we can improve accuracy for individual tasks and enable novel applications such as segmenting all relevant objects in an image through a text prompt, rather than requiring users to laboriously specify the bounding box for each object. We leveraged readily available natural-language labels or descriptions accompanying those datasets and use GPT-4 to harmonize the noisy, unstructured text information with established biomedical object ontologies. We created a large dataset comprising over six million triples of image, segmentation mask, and textual description. On image segmentation, we showed that BiomedParse is broadly applicable, outperforming state-of-the-art methods on 102,855 test image-mask-label triples across 9 imaging modalities (everything). On object detection, which aims to locate a specific object of interest, BiomedParse again attained state-of-the-art performance, especially on objects with irregular shapes (everywhere). On object recognition, which aims to identify all objects in a given image along with their semantic types, we showed that BiomedParse can simultaneously segment and label all biomedical objects in an image (all at once). In summary, BiomedParse is an all-in-one tool for biomedical image analysis by jointly solving segmentation, detection, and recognition for all major biomedical image modalities, paving the path for efficient and accurate image-based biomedical discovery.

Motivation & Objective

• To address the limitations of traditional biomedical image analysis, which treats segmentation, detection, and recognition as separate tasks with high user burden.
• To eliminate the need for user-specified bounding boxes in segmentation by enabling zero-shot, text-guided parsing of all relevant objects in an image.
• To develop a scalable, unified framework for holistic biomedical image analysis that leverages interdependencies between segmentation, detection, and recognition.
• To create a large-scale, multi-modal biomedical image parsing dataset using only standard segmentation datasets and LLM-generated semantic harmonization with biomedical ontologies.

Proposed method

• BiomedParse is pre-trained on a dataset of over 6 million image-mask-description triples constructed by using GPT-4 to align noisy, unstructured natural language descriptions from existing segmentation datasets with established biomedical object ontologies.
• The model employs a multi-task learning framework that jointly optimizes for segmentation, detection, and recognition using a single vision-language backbone, enabling end-to-end parsing from a single text prompt.
• For irregular-shaped objects, the model uses attention map refinement via cross-correlation and iterative shifting to improve mask alignment, especially in 3D modalities like CT and MRI.
• Object recognition is performed through a two-stage process: first, target selection via area-based thresholding (λ × original area) of predicted mask regions; second, non-overlapping mask aggregation using pixel-wise probability ranking.
• The model uses a hierarchical target structure to organize 82 biomedical object types across 9 imaging modalities, enabling modality-specific and anatomically coherent parsing.
• Evaluation metrics for detection include Box Ratio, Convex Ratio, and Inverse Rotational Inertia (IRI), which quantify shape regularity and mask fidelity.

Experimental results

Research questions

• RQ1 Can a single foundation model jointly perform segmentation, detection, and recognition in biomedical images without requiring user-provided bounding boxes?
• RQ2 To what extent can GPT-4 be used to harmonize noisy, unstructured textual descriptions from standard segmentation datasets into structured, ontology-aligned labels for pretraining?
• RQ3 How does joint learning across segmentation, detection, and recognition improve performance compared to task-specific models on diverse biomedical imaging modalities?
• RQ4 Can the model generalize to irregularly shaped objects and detect them more accurately than bounding-box-based methods?
• RQ5 What is the impact of using only standard segmentation datasets with LLM-augmented descriptions on the performance of a unified image parsing model?

Key findings

• BiomedParse outperforms state-of-the-art methods on 102,855 test image-mask-label triples across 9 imaging modalities, demonstrating broad applicability in segmentation.
• The model achieves state-of-the-art performance in object detection, particularly for irregularly shaped objects, as measured by improved Box Ratio, Convex Ratio, and IRI metrics.
• BiomedParse successfully identifies and segments all relevant biomedical objects in an image using only a text prompt, eliminating the need for manual bounding box input.
• The model can detect and reject invalid user inputs describing non-existent objects in the image, demonstrating robustness to semantic hallucinations.
• The use of GPT-4 to align unstructured descriptions with biomedical ontologies enabled the creation of a large-scale dataset of 6.8 million image-mask-description triples from only standard segmentation datasets.
• The two-stage object recognition pipeline effectively reduces mask overlap and improves selection accuracy, with performance superior to baseline methods using Grounding DINO, SAM, and MedSAM.