[Paper Review] Attend and Predict: Understanding Gene Regulation by Selective Attention on Chromatin

Ritambhara Singh, Jack Lanchantin+2 more|arXiv (Cornell University)|Aug 1, 2017

Genomics and Chromatin Dynamics18 references58 citations

TL;DR

AttentiveChrome introduces a hierarchical attention-based deep learning model that predicts gene expression from histone modification signals and provides interpretable insights into which marks and genomic positions drive regulation across many human cell types.

ABSTRACT

The past decade has seen a revolution in genomic technologies that enable a flood of genome-wide profiling of chromatin marks. Recent literature tried to understand gene regulation by predicting gene expression from large-scale chromatin measurements. Two fundamental challenges exist for such learning tasks: (1) genome-wide chromatin signals are spatially structured, high-dimensional and highly modular; and (2) the core aim is to understand what are the relevant factors and how they work together? Previous studies either failed to model complex dependencies among input signals or relied on separate feature analysis to explain the decisions. This paper presents an attention-based deep learning approach; we call AttentiveChrome, that uses a unified architecture to model and to interpret dependencies among chromatin factors for controlling gene regulation. AttentiveChrome uses a hierarchy of multiple Long short-term memory (LSTM) modules to encode the input signals and to model how various chromatin marks cooperate automatically. AttentiveChrome trains two levels of attention jointly with the target prediction, enabling it to attend differentially to relevant marks and to locate important positions per mark. We evaluate the model across 56 different cell types (tasks) in human. Not only is the proposed architecture more accurate, but its attention scores also provide a better interpretation than state-of-the-art feature visualization methods such as saliency map. Code and data are shared at www.deepchrome.org

Motivation & Objective

• Understand how chromatin marks regulate gene expression across cell types using a unified predictive model.
• Model spatially structured, high-dimensional chromatin signals while capturing modular interactions among marks.
• Provide interpretable mechanisms to identify which marks and genomic positions influence gene expression.

Proposed method

• Use a hierarchy of bidirectional LSTMs to encode bin-level signals for each histone modification (HM).
• Apply bin-level soft attention to identify important genomic positions within each HM mark.
• Use a second LSTM (HM-level) to model interactions among HM marks and apply HM-level soft attention to weigh marks by their importance.
• Train the entire architecture end-to-end with a binary gene-expression prediction objective (high vs low).
• Compare AttentiveChrome to baselines (DeepChrome, LSTM, CNN-based models) and assess interpretability via attention weights.
• Validate interpretability by correlating attention maps with independent HM activity data not used in training.

Experimental results

Research questions

• RQ1 Can an attention-based hierarchical model jointly learn dependencies among chromatin marks and their positional effects to predict gene expression?
• RQ2 Do bin-level and HM-level attentions yield biologically meaningful interpretations of regulatory mechanisms?
• RQ3 How does AttentiveChrome perform relative to state-of-the-art baselines across many cell types, and can its attention provide better interpretability than saliency-based methods?

Key findings

• AttentiveChrome achieves higher predictive performance than several baselines across 56 cell types (AUC-based evaluation).
• Two-level attention (bin-level and HM-level) enables automatic identification of relevant genomic positions and influential histone marks.
• Attention weights correlate more strongly with independent activity signals than saliency maps or class-optimization visualizations, supporting interpretability.
• Model demonstrates effective handling of modular, structured inputs by treating each HM as a module and learning their interactions.
• Qualitative analyses show biologically coherent patterns, e.g., promoter marks concentrate near TSS for active genes and repressor marks are more influential for OFF genes.