[Paper Review] Reusability report: Prostate cancer stratification with diverse biologically-informed neural architectures

Christian Pedersen, Tiberiu Teşileanu+5 more|arXiv (Cornell University)|Sep 28, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This study validates the reproducibility and robustness of P-NET, a biologically-informed deep neural network for prostate cancer stratification, demonstrating that Reactome pathway knowledge is essential for its performance. It further reveals that distinct neural architectures—P-NET and graph neural networks—produce divergent, persistent per-patient predictions despite similar overall accuracy, highlighting architecture-specific biases in clinical AI models.

ABSTRACT

In Elmarakeby et al., "Biologically informed deep neural network for prostate cancer discovery", a feedforward neural network with biologically informed, sparse connections (P-NET) was presented to model the state of prostate cancer. We verified the reproducibility of the study conducted by Elmarakeby et al., using both their original codebase, and our own re-implementation using more up-to-date libraries. We quantified the contribution of network sparsification by Reactome biological pathways, and confirmed its importance to P-NET's superior performance. Furthermore, we explored alternative neural architectures and approaches to incorporating biological information into the networks. We experimented with three types of graph neural networks on the same training data, and investigated the clinical prediction agreement between different models. Our analyses demonstrated that deep neural networks with distinct architectures make incorrect predictions for individual patient that are persistent across different initializations of a specific neural architecture. This suggests that different neural architectures are sensitive to different aspects of the data, an important yet under-explored challenge for clinical prediction tasks.

Motivation & Objective

• To verify the reproducibility of P-NET, a biologically-informed neural network for prostate cancer stratification, using both original and refactored codebases.
• To assess the contribution of Reactome pathway information to model performance by comparing biologically-informed sparsification with random sparsification.
• To evaluate alternative neural architectures—specifically three graph neural networks—on the same dataset to compare predictive performance and consistency.
• To investigate per-patient prediction disparities across different architectures, focusing on persistent errors across model initializations.
• To explore the clinical implications of architectural design in interpretable machine learning for cancer genomics.

Proposed method

• Reimplemented the original P-NET model in PyTorch to improve code reusability and ensure reproducibility with modern deep learning libraries.
• Trained 30 randomly initialized instances of P-NET and three GNN variants (GCN, GAT, GraphSAGE) on the same training set of 807 prostate cancer patients.
• Used fixed sparsity levels in P-NET to isolate the impact of biologically informed pathway structure by comparing with randomly permuted pathway assignments.
• Evaluated models on a held-out test set of 102 patients using standard metrics: AUC, AUPR, F1, precision, recall, and accuracy.
• Computed correlation matrices between model predictions to quantify agreement across architectures and analyzed per-patient prediction discrepancies.
• Applied statistical tests (Kolmogorov-Smirnov, FDR-corrected) to assess significance of prediction differences between models.

Experimental results

Research questions

• RQ1 Is the original P-NET model for prostate cancer stratification reproducible using the released code and a modern PyTorch implementation?
• RQ2 To what extent does the inclusion of biologically informed Reactome pathways contribute to P-NET’s performance, independent of model sparsity?
• RQ3 How do graph neural networks with different inductive biases compare to P-NET in predicting metastatic prostate cancer on the same dataset?
• RQ4 Are there persistent per-patient prediction errors across different initializations of the same neural architecture, indicating architectural bias?
• RQ5 To what extent do different neural architectures produce divergent predictions for the same patient, and what are the implications for clinical decision-making?

Key findings

• The PyTorch reimplementation of P-NET reproduced the original results with high fidelity, confirming the reproducibility of the core model.
• Randomly permuted pathway assignments—preserving sparsity but removing biological structure—caused a significant drop in performance, proving that biological knowledge is critical to P-NET’s success.
• Graph neural networks (GCN, GAT, GraphSAGE) showed slightly lower overall performance than P-NET, with AUPR values of 0.86 ± 0.01 compared to P-NET’s 0.89 ± 0.01.
• The correlation between P-NET and GNN predictions was low (average Pearson correlation = 0.682), indicating that models with different architectures focus on different aspects of the data.
• For 50 out of 102 test patients, the predicted metastatic probability differed by at least 10% between GAT and P-NET, and these differences were statistically significant (FDR < 0.05).
• Incorrect predictions were not random but persisted across multiple weight initializations of the same architecture, indicating that architectural design induces systematic, non-random biases in clinical predictions.