[Paper Review] BabyNet: Residual Transformer Module for Birth Weight Prediction on Fetal Ultrasound Video

Szymon Płotka, Michal K. Grzeszczyk+5 more|arXiv (Cornell University)|May 19, 2022

Neonatal and fetal brain pathologyMedicine19 references17 citations

TL;DR

BabyNet is an end-to-end deep learning framework that uses a novel Residual Transformer Module (RTM) to directly predict fetal birth weight from 2D + t fetal ultrasound video scans, bypassing the need for standard plane detection. It outperforms state-of-the-art methods and achieves accuracy comparable to expert clinicians, with the best performance when combining human and model predictions (mMAPE: 5.2%).

ABSTRACT

Predicting fetal weight at birth is an important aspect of perinatal care, particularly in the context of antenatal management, which includes the planned timing and the mode of delivery. Accurate prediction of weight using prenatal ultrasound is challenging as it requires images of specific fetal body parts during advanced pregnancy which is difficult to capture due to poor quality of images caused by the lack of amniotic fluid. As a consequence, predictions which rely on standard methods often suffer from significant errors. In this paper we propose the Residual Transformer Module which extends a 3D ResNet-based network for analysis of 2D+t spatio-temporal ultrasound video scans. Our end-to-end method, called BabyNet, automatically predicts fetal birth weight based on fetal ultrasound video scans. We evaluate BabyNet using a dedicated clinical set comprising 225 2D fetal ultrasound videos of pregnancies from 75 patients performed one day prior to delivery. Experimental results show that BabyNet outperforms several state-of-the-art methods and estimates the weight at birth with accuracy comparable to human experts. Furthermore, combining estimates provided by human experts with those computed by BabyNet yields the best results, outperforming either of other methods by a significant margin. The source code of BabyNet is available at https://github.com/SanoScience/BabyNet.

Motivation & Objective

• To develop an end-to-end deep learning method for direct fetal birth weight prediction from 2D + t ultrasound video scans.
• To address the limitations of traditional methods that rely on manual identification of standard fetal planes and heuristic formulae.
• To improve prediction accuracy by integrating 3D Multi-Head Self-Attention with residual connections and temporal position encoding.
• To evaluate the model on a clinical dataset collected one day prior to delivery, ensuring real-world relevance.
• To explore the potential of combining human expert estimates with AI predictions for superior performance.

Proposed method

• BabyNet is built on a 3D ResNet-18 backbone, replacing the last two residual modules with a novel Residual Transformer Module (RTM).
• The RTM integrates 3D Multi-Head Self-Attention (MHSA) with 3D convolutional layers and batch normalization for joint local and global feature learning.
• Temporal position encoding (TPE) is incorporated into the MHSA mechanism to preserve spatio-temporal order in the video input.
• The network processes 16-frame video segments, applies global average pooling, and outputs a single birth weight prediction per patient.
• The model is trained end-to-end using a dedicated clinical dataset of 225 videos from 75 patients scanned one day before delivery.
• Performance is evaluated using 5-fold cross-validation with metrics: mean absolute error (mMAE), root mean square error (mRMSE), and mean absolute percentage error (mMAPE).

Experimental results

Research questions

• RQ1 Can a hybrid CNN-Transformer architecture effectively predict fetal birth weight directly from 2D + t ultrasound video scans without requiring standard plane detection?
• RQ2 Does the integration of a Residual Transformer Module with temporal position encoding improve prediction accuracy compared to standard 3D CNNs or pure Transformers?
• RQ3 How does the performance of BabyNet compare to expert clinicians using standard clinical tools?
• RQ4 Can combining human expert predictions with BabyNet’s output yield significantly better results than either method alone?
• RQ5 Does the model generalize well on a small, clinically acquired dataset collected shortly before delivery?

Key findings

• BabyNet achieved a mean absolute error (mMAE) of 254 ± 230 g, root mean square error (mRMSE) of 341 ± 215 g, and mean absolute percentage error (mMAPE) of 7.5 ± 6.6% on the test set.
• The combination of clinicians’ estimates and BabyNet’s predictions achieved the best performance with mMAE of 180 ± 156 g, mRMSE of 237 ± 145 g, and mMAPE of 5.2 ± 4.6%.
• BabyNet’s performance was not statistically different from expert clinicians reported in prior work (p-value = 0.6), though it outperformed clinicians in a separate comparison (p-value = 0.07).
• The ablation study confirmed that adding the RTM and temporal position encoding significantly improved performance over the base 3D ResNet-18.
• The hybrid RTM design, combining 3D convolutions with MHSA and TPE, yielded better results than pure CNNs, pure Transformers, or CNN+Transformer hybrids.
• The model demonstrated that end-to-end learning from raw video clips can match expert-level accuracy, suggesting strong clinical utility as a decision-support tool.