[Paper Review] IGLOO: Slicing the Features Space to Represent Sequences

Vsevolod Sourkov|arXiv (Cornell University)|Jul 9, 2018

Image Processing and 3D Reconstruction4 citations

TL;DR

IGLOO introduces a novel neural network architecture that represents sequences by analyzing relationships between non-local, spatially sliced patches from feature maps of stacked 1D convolutions. It achieves competitive performance on long sequences (up to 20,000 steps), matching Transformers on Wikitext-2 (perplexity ~20.5) and setting a new SOTA of 98.46% accuracy on a long non-coding RNA classification task, with reduced memory usage due to sparse patch-based attention.

ABSTRACT

Historically, Recurrent neural networks (RNNs) and its variants such as LSTM and GRU and more recently Transformers have been the standard go-to components when processing sequential data with neural networks. One notable issue is the relative difficulty to deal with long sequences (i.e. more than 20,000 steps). We introduce IGLOO, a new neural network architecture which aims at being efficient for short sequences but also at being able to deal with long sequences. IGLOOs core idea is to use the relationships between non-local patches sliced out of the features maps of successively applied convolutions to build a representation for the sequence. We show that the model can deal with dependencies of more than 20,000 steps in a reasonable time frame. We stress test IGLOO on the copy-memory and addition tasks, as well as permuted MNIST (98.4%). For a larger task we apply this new structure to the Wikitext-2 dataset Merity et al. (2017b) and achieve a perplexity in line with baseline Transformers but lower than baseline AWD-LSTM. We also present how IGLOO is already used today in production for bioinformatics tasks.

Motivation & Objective

• To address the challenge of modeling long sequential dependencies beyond 20,000 time steps, where standard RNNs and Transformers face computational and memory bottlenecks.
• To develop a lightweight, efficient alternative to Transformers that reduces O(L²) complexity while preserving long-range dependency modeling.
• To enable practical deployment of sequence models in resource-constrained domains such as bioinformatics, where sequences can exceed 3,000 nucleotides.
• To explore a new inductive bias based on non-local patch correlations in feature maps, distinct from self-attention or recurrence.
• To demonstrate that sparse, structured relationships between patches can yield competitive performance on standard benchmarks and real-world tasks.

Proposed method

• IGLOO-base uses causal 1D convolutions to extract feature maps from input sequences, followed by gathering p non-local patches from the time dimension using a learnable or stochastic sampling strategy.
• Each patch is a contiguous segment of p time steps from the feature map, and J such patches are processed in parallel to form a compact representation of size J × K, where K is the number of convolutional filters.
• The patch selection is guided by a Gaussian distribution centered on the current time step, enabling focus on local and non-local context with tunable variance.
• IGLOO-seq replaces the self-attention mechanism in Transformers with an IGLOO-base block, maintaining the same architecture but reducing computational complexity via sparse patch interactions.
• The model uses a stack of two 1D convolutional layers with kernel size 3 and causal padding, followed by patch gathering and global average pooling for sequence-level representation.
• The method is designed to be memory-efficient by limiting the number of patches J, which scales sub-quadratically with sequence length L, unlike the O(L²) attention in Transformers.

Experimental results

Research questions

• RQ1 Can a patch-based correlation mechanism between non-local regions of a feature map effectively model long-range dependencies in sequences?
• RQ2 Does replacing self-attention with IGLOO-base in a Transformer-like architecture maintain competitive performance while reducing memory and computation costs?
• RQ3 Can IGLOO-scale to real-world long-sequence tasks such as RNA sequence classification, where sequences exceed 3,000 tokens?
• RQ4 How does the number of patches J affect model performance and memory efficiency in long-sequence modeling?
• RQ5 Is IGLOO robust to stochastic patch sampling, given that patch locations are randomly sampled per time step?

Key findings

• IGLOO-seq achieves a test perplexity of 20.5 on Wikitext-2, matching the performance of standard Transformers while using less memory.
• The model maintains consistent performance across different patch counts (J=50 and J=100), with only a 0.5 perplexity variance, indicating robustness to hyperparameter choice.
• On the FEELnc dataset, IGLOO-base achieved a state-of-the-art accuracy of 98.46% in classifying coding potential of long non-coding RNAs, outperforming prior RNN and Transformer baselines.
• The model successfully captures dependencies over sequences of up to 20,000 time steps, demonstrating scalability beyond typical RNN and Transformer limits.
• With J=600 patches on 3,000-length RNA sequences, IGLOO-base achieves high performance with J << L, proving efficiency gains in real-world bioinformatics applications.
• The model shows low variance in predictions across random patch sampling, suggesting stable optimization and generalization despite stochastic patch selection.