[Paper Review] A Deep-Generative Hybrid Model to Integrate Multimodal and Dynamic Connectivity for Predicting Spectrum-Level Deficits in Autism

Niharika Shimona D’Souza, Mary Beth Nebel+5 more|arXiv (Cornell University)|Jul 3, 2020

Advanced Neuroimaging Techniques and Applications31 references4 citations

TL;DR

This paper proposes a deep-generative hybrid model that integrates dynamic functional MRI (rs-fMRI) connectivity and diffusion tensor imaging (DTI) tractography to predict multidimensional clinical severity in Autism Spectrum Disorder (ASD). Using a structurally-regularized dynamic dictionary learning (sr-DDL) to extract anatomically informed functional networks and an LSTM-ANN to model temporal dynamics and predict severity, the framework achieves state-of-the-art performance in cross-validated multi-score prediction, with improved generalization and interpretable neural signatures.

ABSTRACT

We propose an integrated deep-generative framework, that jointly models complementary information from resting-state functional MRI (rs-fMRI) connectivity and diffusion tensor imaging (DTI) tractography to extract predictive biomarkers of a disease. The generative part of our framework is a structurally-regularized Dynamic Dictionary Learning (sr-DDL) model that decomposes the dynamic rs-fMRI correlation matrices into a collection of shared basis networks and time varying patient-specific loadings. This matrix factorization is guided by the DTI tractography matrices to learn anatomically informed connectivity profiles. The deep part of our framework is an LSTM-ANN block, which models the temporal evolution of the patient sr-DDL loadings to predict multidimensional clinical severity. Our coupled optimization procedure collectively estimates the basis networks, the patient-specific dynamic loadings, and the neural network weights. We validate our framework on a multi-score prediction task in 57 patients diagnosed with Autism Spectrum Disorder (ASD). Our hybrid model outperforms state-of-the-art baselines in a five-fold cross validated setting and extracts interpretable multimodal neural signatures of brain dysfunction in ASD.

Motivation & Objective

• To develop a unified framework that integrates multimodal neuroimaging (rs-fMRI and DTI) and dynamic connectivity for predicting continuous clinical severity in ASD.
• To address the limitations of existing methods that focus on static connectomes or single modalities, especially in predicting continuous behavioral deficits.
• To incorporate anatomical priors from DTI into the dynamic decomposition of rs-fMRI to improve interpretability and biological plausibility.
• To model temporal evolution of functional connectivity patterns using an LSTM-ANN to enhance prediction of multidimensional clinical scores.

Proposed method

• The framework uses a structurally-regularized Dynamic Dictionary Learning (sr-DDL) model to decompose time-varying rs-fMRI correlation matrices into shared basis networks (B) and patient-specific dynamic loadings (c^t_n), guided by DTI tractography matrices.
• The sr-DDL objective minimizes reconstruction error of correlation matrices under a weighted norm defined by DTI connectivity, enforcing anatomical plausibility.
• Patient-specific dynamic loadings (c^t_n) are fed into an LSTM-ANN block that models temporal trends and predicts M-dimensional clinical severity scores.
• A joint optimization procedure simultaneously learns the basis networks (B), dynamic loadings (c^t_n), and deep neural network weights (Θ) via a coupled objective function.
• The model employs attention mechanisms in the A-ANN to identify time intervals most relevant for prediction, enhancing interpretability.
• The framework is trained and validated on 57 ASD patients using five-fold cross-validation, with performance evaluated via median absolute error (MAE) and mutual information (MI).

Experimental results

Research questions

• RQ1 Can a hybrid deep-generative model effectively integrate dynamic rs-fMRI connectivity and structural DTI information to predict continuous clinical severity in ASD?
• RQ2 How does incorporating DTI tractography as an anatomical prior improve the interpretability and accuracy of functional network decomposition in rs-fMRI?
• RQ3 To what extent does modeling temporal dynamics in functional connectivity enhance prediction of multidimensional ASD symptom severity compared to static models?
• RQ4 Can the learned dynamic loadings and attention mechanisms in the deep network reveal biologically meaningful, patient-specific patterns of brain dysfunction in ASD?
• RQ5 Does the joint optimization of generative and deep learning components lead to better generalization on unseen patients than separate or baseline models?

Key findings

• The proposed hybrid model achieved the lowest median absolute error (MAE) on test data across all clinical scores—2.84 for ADOS, 17.81 for SRS, and 13.50 for Praxis—outperforming all baselines.
• The model achieved the highest mutual information (MI) on test data (0.34 for ADOS, 0.88 for SRS, 0.85 for Praxis), indicating stronger predictive relationships with clinical severity.
• The sr-DDL model successfully extracted four interpretable subnetworks: one linked to the default mode network (DMN), one to visual-sensorimotor regions, one to central executive and insular networks, and one to prefrontal and subcortical regions involved in social-emotional regulation.
• Attention mechanisms in the A-ANN highlighted different time intervals (early vs. late scan) as most predictive, reflecting patient-specific heterogeneity in dynamic connectivity patterns.
• The model demonstrated superior generalization, with test performance closely following the ideal diagonal line in prediction plots, unlike baselines that overfit on individual scores.
• Ablation studies showed that removing DTI regularization led to higher test MAE (3.27 for ADOS) and lower MI (0.26), confirming the value of structural priors in improving robustness and interpretability.