[Paper Review] OTFPF: Optimal Transport-Based Feature Pyramid Fusion Network for Brain Age Estimation with 3D Overlapped ConvNeXt

Fu Yu, Yanyan Huang+7 more|arXiv (Cornell University)|May 10, 2022

Brain Tumor Detection and Classification4 citations

TL;DR

This paper proposes OTFPF, an end-to-end deep learning framework for brain age estimation using 3D T1-weighted MRIs, integrating optimal transport-based feature pyramid fusion, 3D overlapped ConvNeXt, and multi-pathway feature fusion to enhance semi-multimodal and multi-scale feature learning. The model achieves state-of-the-art performance with a mean absolute error (MAE) of 2.097, Pearson correlation (PCC) of 0.993, and Spearman’s rank correlation (SRCC) of 0.989 on 11,728 subjects aged 3–97.

ABSTRACT

Chronological age of healthy brain is able to be predicted using deep neural networks from T1-weighted magnetic resonance images (T1 MRIs), and the predicted brain age could serve as an effective biomarker for detecting aging-related diseases or disorders. In this paper, we propose an end-to-end neural network architecture, referred to as optimal transport based feature pyramid fusion (OTFPF) network, for the brain age estimation with T1 MRIs. The OTFPF consists of three types of modules: Optimal Transport based Feature Pyramid Fusion (OTFPF) module, 3D overlapped ConvNeXt (3D OL-ConvNeXt) module and fusion module. These modules strengthen the OTFPF network's understanding of each brain's semi-multimodal and multi-level feature pyramid information, and significantly improve its estimation performances. Comparing with recent state-of-the-art models, the proposed OTFPF converges faster and performs better. The experiments with 11,728 MRIs aged 3-97 years show that OTFPF network could provide accurate brain age estimation, yielding mean absolute error (MAE) of 2.097, Pearson's correlation coefficient (PCC) of 0.993 and Spearman's rank correlation coefficient (SRCC) of 0.989, between the estimated and chronological ages. Widespread quantitative experiments and ablation experiments demonstrate the superiority and rationality of OTFPF network. The codes and implement details will be released on GitHub: https://github.com/ZJU-Brain/OTFPF after final decision.

Motivation & Objective

• To improve brain age estimation accuracy by leveraging multi-scale and semi-multimodal features from T1, gray matter, and white matter MRIs.
• To address limitations in existing models that underutilize feature hierarchy and fail to optimally fuse multi-modal inputs.
• To develop an end-to-end architecture that integrates optimal transport for structured feature fusion and improves convergence speed.
• To validate the importance of biological sex as a modality in brain age prediction.
• To demonstrate superior performance over recent state-of-the-art models in both accuracy and training efficiency.

Proposed method

• The OTFPF network employs a multi-pathway architecture: one backbone for T1 MRI, two branches for gray and white matter images, and a separate path for sex label.
• It uses 3D overlapped ConvNeXt (3D OL-ConvNeXt) modules to extract hierarchical features with reduced computational cost and improved local feature capture.
• Optimal Transport-based Feature Pyramid Fusion (OTFPF) module integrates multi-scale features across pathways using optimal transport embedding to learn structured, differentiable feature alignment.
• The fusion module cascades feature fusion at four stages, progressively combining semi-multimodal features with the backbone pathway to enrich representation.
• The architecture is end-to-end trainable, with loss minimized via mean absolute error (MAE) and correlation-based metrics.
• Ablation studies validate the contribution of each component, including the sex label, OTEM, and 3D OL-ConvNeXt design.

Experimental results

Research questions

• RQ1 Can optimal transport-based feature fusion improve the integration of multi-scale and semi-multimodal features in brain age estimation?
• RQ2 How does the 3D overlapped ConvNeXt architecture compare to standard 3D ConvNeXt in capturing brain morphological patterns?
• RQ3 What is the contribution of biological sex as a modality in improving estimation accuracy?
• RQ4 Does end-to-end training with joint optimization of multi-pathway features lead to faster convergence and better performance?
• RQ5 How do architectural components like OTEM and FPFN contribute to the model’s robustness and accuracy?

Key findings

• The OTFPF network achieves a mean absolute error (MAE) of 2.097, the lowest among all compared models, indicating high estimation accuracy.
• The model achieves a Pearson correlation coefficient (PCC) of 0.993 and a Spearman’s rank correlation coefficient (SRCC) of 0.989, demonstrating strong alignment between estimated and chronological ages.
• Ablation studies confirm that removing the optimal transport embedding (OTEM) increases MAE to 2.211, highlighting its critical role in feature fusion.
• Excluding the sex label degrades performance to an MAE of 2.293, underscoring its importance in modeling sex-specific aging patterns.
• Replacing 3D OL-ConvNeXt with standard 3D ConvNeXt increases MAE to 2.718, proving the superiority of the overlapped convolution design in feature extraction.
• The model converges faster and outperforms SFCN, GLTN, and TSAN across all metrics, establishing a new state-of-the-art on the tested dataset.