[Paper Review] DeepGene Transformer: Transformer for the gene expression-based classification of cancer subtypes

Anwar A. Khan, Boreom Lee|arXiv (Cornell University)|Aug 26, 2021

Machine Learning in Bioinformatics17 citations

TL;DR

DeepGene Transformer proposes a novel end-to-end deep learning model based on the Transformer architecture to classify cancer subtypes directly from high-dimensional gene expression data, leveraging multi-head self-attention to identify relevant biomarkers without prior feature selection. It outperforms traditional and state-of-the-art methods in classification accuracy across multiple cancer datasets, demonstrating superior performance and robustness in molecular subtyping.

ABSTRACT

Cancer and its subtypes constitute approximately 30% of all causes of death globally and display a wide range of heterogeneity in terms of clinical and molecular responses to therapy. Molecular subtyping has enabled the use of precision medicine to overcome these challenges and provide significant biological insights to predict prognosis and improve clinical decision-making. Over the past decade, conventional machine learning (ML) and deep learning (DL) algorithms have been widely espoused for the classification of cancer subtypes from gene expression datasets. However, these methods are potentially biased toward the identification of cancer biomarkers. Hence, an end-to-end deep learning approach, DeepGene Transformer, is proposed which addresses the complexity of high-dimensional gene expression with a multi-head self-attention module by identifying relevant biomarkers across multiple cancer subtypes without requiring feature selection as a pre-requisite for the current classification algorithms. Comparative analysis reveals that the proposed DeepGene Transformer outperformed the commonly used traditional and state-of-the-art classification algorithms and can be considered an efficient approach for classifying cancer and its subtypes, indicating that any improvement in deep learning models in computational biologists can be reflected well in this domain as well.

Motivation & Objective

• To address the challenge of classifying heterogeneous cancer subtypes from high-dimensional gene expression data using an end-to-end deep learning approach.
• To reduce bias toward pre-selected biomarkers by eliminating the need for explicit feature selection in classification pipelines.
• To develop a model that automatically identifies relevant biomarkers across multiple cancer subtypes through attention mechanisms.
• To improve classification performance over conventional machine learning and deep learning models in cancer molecular subtyping.
• To demonstrate the effectiveness of the Transformer architecture in genomics applications, particularly for precision medicine.

Proposed method

• The model employs a multi-head self-attention mechanism to capture complex dependencies in high-dimensional gene expression data.
• It processes raw gene expression profiles directly without requiring dimensionality reduction or feature selection.
• The architecture uses positional encoding to preserve the order of genes as input tokens.
• A feed-forward network is applied after the self-attention layer to refine representations.
• The final classification head produces subtype predictions using a softmax layer on the [CLS] token representation.
• The model is trained end-to-end using cross-entropy loss and optimized via AdamW with learning rate scheduling.

Experimental results

Research questions

• RQ1 Can a Transformer-based model achieve superior classification performance on cancer subtypes without relying on feature selection?
• RQ2 How well does the self-attention mechanism identify relevant biomarkers across diverse cancer subtypes in gene expression data?
• RQ3 Does the proposed DeepGene Transformer outperform established machine learning and deep learning baselines in cancer subtyping tasks?
• RQ4 To what extent does the model generalize across different cancer types and datasets?
• RQ5 Can attention weights provide biologically interpretable insights into the molecular drivers of cancer subtypes?

Key findings

• DeepGene Transformer achieved higher classification accuracy than traditional machine learning and state-of-the-art deep learning models on multiple cancer datasets.
• The model demonstrated robust performance across diverse cancer types, including those with high molecular heterogeneity.
• The self-attention mechanism effectively identified biologically relevant genes associated with cancer subtypes without prior feature selection.
• Comparative analysis confirmed that the model outperformed models like SVM, Random Forest, and ResNet-based architectures in AUC and F1-score metrics.
• Attention weights provided interpretable patterns, highlighting key genes and pathways linked to specific subtypes.
• The end-to-end training framework eliminated the need for preprocessing steps such as feature selection, streamlining the classification pipeline.