[Paper Review] Evaluating deep variational autoencoders trained on pan-cancer gene expression

Gregory P. Way, Casey S. Greene|arXiv (Cornell University)|Nov 13, 2017

Molecular Biology Techniques and Applications19 references17 citations

TL;DR

This study evaluates deep variational autoencoders (VAEs) trained on pan-cancer RNA-seq data from The Cancer Genome Atlas (TCGA), comparing one- and two-hidden layer VAEs with PCA, ICA, NMF, and ADAGE for dimensionality reduction. While deeper VAEs offer only modest performance gains, VAE latent spaces reveal biologically meaningful patterns, particularly in high-grade serous ovarian cancer (HGSC) subtype comparisons, where they identify collagen catabolic processes linked to mesenchymal subtypes.

ABSTRACT

Cancer is a heterogeneous disease with diverse molecular etiologies and outcomes. The Cancer Genome Atlas (TCGA) has released a large compendium of over 10,000 tumors with RNA-seq gene expression measurements. Gene expression captures the diverse molecular profiles of tumors and can be interrogated to reveal differential pathway activations. Deep unsupervised models, including Variational Autoencoders (VAE) can be used to reveal these underlying patterns. We compare a one-hidden layer VAE to two alternative VAE architectures with increased depth. We determine the additional capacity marginally improves performance. We train and compare the three VAE architectures to other dimensionality reduction techniques including principal components analysis (PCA), independent components analysis (ICA), non-negative matrix factorization (NMF), and analysis of gene expression by denoising autoencoders (ADAGE). We compare performance in a supervised learning task predicting gene inactivation pan-cancer and in a latent space analysis of high grade serous ovarian cancer (HGSC) subtypes. We do not observe substantial differences across algorithms in the classification task. VAE latent spaces offer biological insights into HGSC subtype biology.

Motivation & Objective

• To assess the performance of deep variational autoencoders (VAEs) with increasing depth on pan-cancer gene expression data.
• To compare VAEs against established dimensionality reduction techniques (PCA, ICA, NMF, ADAGE) in supervised and unsupervised tasks.
• To evaluate whether VAE latent spaces provide biologically meaningful representations, especially in detecting molecular subtypes of high-grade serous ovarian cancer (HGSC).
• To investigate the utility of latent space arithmetic in uncovering disease-relevant biological pathways from gene expression data.

Proposed method

• Trained three VAE architectures: one-hidden layer (Tybalt), and two-hidden layer models with 100 and 300 units, using Keras with TensorFlow backend.
• Applied grid search over hyperparameters (learning rate, batch size, epochs, kappa for KL warmup) to minimize holdout validation loss.
• Used zero-one normalization for VAE, NMF, and ADAGE; z-score normalization for PCA and ICA on log2(FPKM+1) TCGA RNA-seq data from 10,459 samples.
• Evaluated models via 5-fold cross-validation on predicting NF1 inactivation using elastic net logistic regression.
• Performed latent space arithmetic by subtracting mean vector representations of mesenchymal and immunoreactive HGSC subtypes to identify differentiating features.
• Conducted pathway overrepresentation analysis (ORA) on high-weight genes from subtraction results using Gene Ontology (GO) terms.

Experimental results

Research questions

• RQ1 How does increasing the depth of a VAE architecture affect performance in capturing biologically relevant patterns in pan-cancer gene expression?
• RQ2 How do VAEs compare to traditional dimensionality reduction methods (PCA, ICA, NMF, ADAGE) in predicting NF1 inactivation across cancers?
• RQ3 Can latent space arithmetic in VAEs reveal biologically meaningful differences between high-grade serous ovarian cancer (HGSC) subtypes?
• RQ4 Which biological pathways are most strongly differentiated in the VAE latent space between mesenchymal and immunoreactive HGSC subtypes?

Key findings

• Deeper VAE architectures (two hidden layers) showed only modest improvements in performance over the one-hidden layer Tybalt model, with minimal gains in validation loss.
• In predicting pan-cancer NF1 inactivation, raw gene expression features performed best (AUROC = 68.4%), followed by PCA (AUROC = 65.6%), while VAEs performed comparably to other dimensionality reduction methods.
• The shuffled gene expression control dataset achieved AUROC = 55.4%, suggesting potential artifacts from low sample size in classification tasks.
• VAE-based latent spaces, particularly Tybalt and the 300-unit two-hidden-layer VAE, identified 'collagen catabolic process' as the most significantly overrepresented GO term (p = 1.8e-09 and 1.7e-03, respectively) in HGSC subtype comparisons.
• Linear methods (PCA, ICA, NMF) showed limited or no significant pathway enrichment, with NMF identifying 'homophilic cell adhesion' (p = 3.3e-06) as the only significant term.
• ADAGE and ICA did not identify any significant pathways in the ORA analysis, underscoring the unique biological insight from nonlinear VAE representations.