[Paper Review] 3D Medical Point Transformer: Introducing Convolution to Attention Networks for Medical Point Cloud Analysis

Jianhui Yu, Chaoyi Zhang+6 more|arXiv (Cornell University)|Dec 9, 2021

Radiomics and Machine Learning in Medical Imaging22 citations

TL;DR

This paper proposes 3DMedPT, a novel Transformer-based model for 3D medical point cloud analysis that integrates convolutional feature summarization and Multi-Graph Reasoning to enhance local and global feature learning. It achieves state-of-the-art performance on the IntrA benchmark for classification and segmentation, with strong generalization on general 3D point cloud datasets like ModelNet40 and ShapeNetPart.

ABSTRACT

General point clouds have been increasingly investigated for different tasks, and recently Transformer-based networks are proposed for point cloud analysis. However, there are barely related works for medical point clouds, which are important for disease detection and treatment. In this work, we propose an attention-based model specifically for medical point clouds, namely 3D medical point Transformer (3DMedPT), to examine the complex biological structures. By augmenting contextual information and summarizing local responses at query, our attention module can capture both local context and global content feature interactions. However, the insufficient training samples of medical data may lead to poor feature learning, so we apply position embeddings to learn accurate local geometry and Multi-Graph Reasoning (MGR) to examine global knowledge propagation over channel graphs to enrich feature representations. Experiments conducted on IntrA dataset proves the superiority of 3DMedPT, where we achieve the best classification and segmentation results. Furthermore, the promising generalization ability of our method is validated on general 3D point cloud benchmarks: ModelNet40 and ShapeNetPart. Code is released.

Motivation & Objective

• Address the challenge of analyzing complex, irregular, and topology-diverse 3D medical point clouds with incomplete pathological structures.
• Overcome the limitations of existing MLP-based models (e.g., PointNet) in capturing long-range dependencies and expressive shape features in medical data.
• Improve feature learning in low-data regimes typical of medical datasets by introducing positional embeddings and Multi-Graph Reasoning.
• Design a model that performs well on medical point clouds while generalizing effectively to general 3D point cloud benchmarks.

Proposed method

• Integrate convolutional layers within the self-attention mechanism to summarize local context and generate holistic geometric features.
• Augment query features with local neighborhood context before attention computation to enhance context awareness.
• Apply learnable positional embeddings at query, key, and value projections to model precise local geometry in irregular medical point clouds.
• Propose Multi-Graph Reasoning (MGR) to learn multiple channel graphs with learnable adjacency matrices, enabling global feature propagation across channels.
• Use softmax normalization in the attention mechanism for optimal feature summarization, validated empirically.
• Train the model end-to-end on medical and general 3D datasets with data augmentation and unit cube normalization.

Experimental results

Research questions

• RQ1 Can a Transformer-based architecture effectively model complex and diverse anatomical topologies in 3D medical point clouds?
• RQ2 How can local geometric context and global feature interactions be jointly optimized in low-shot medical point cloud learning?
• RQ3 To what extent can positional embeddings and Multi-Graph Reasoning improve feature representation in medical point cloud analysis?
• RQ4 Does the proposed model generalize well to non-medical 3D point cloud benchmarks like ModelNet40 and ShapeNetPart?

Key findings

• 3DMedPT achieves the highest F1 score of 0.922 on the IntrA classification benchmark, outperforming all baselines.
• On the IntrA segmentation task, the model produces highly precise results, with only minor errors in complex or small-structure cases.
• The model achieves 90.9% accuracy on RetinalOCT (converted to 2D point clouds), outperforming PointNet and DGCNN despite data conversion limitations.
• On AdrenalMNIST3D, 3DMedPT reaches 79.1% accuracy, ranking second among point-based models and approaching AutoKeras performance.
• The model generalizes well to general 3D point cloud datasets, achieving strong results on ModelNet40 and ShapeNetPart.
• Extensive ablation shows that softmax normalization yields the best attention performance, and Ck=32 with h=8 provides optimal attention head configuration.