[Paper Review] IGLOO: Slicing the Features Space to Represent Long Sequences.

Vsevolod Sourkov|arXiv (Cornell University)|Jul 9, 2018

Topic Modeling29 references3 citations

TL;DR

IGLOO is a novel neural network architecture that slices feature maps from stacked convolutions to model long-range dependencies in sequences, enabling efficient processing of sequences over 20,000 steps. It achieves competitive performance on long sequence tasks like copy-memory, addition, permuted MNIST (98.4%), and Wikitext-2, matching Transformer perplexity while outperforming AWD-LSTM.

ABSTRACT

Historically, Recurrent neural networks (RNNs) and its variants such as LSTM and GRU and more recently Transformers have been the standard go-to components when processing sequential data with neural networks. One notable issue is the relative difficulty to deal with long sequences (i.e. more than 20,000 steps). We introduce IGLOO, a new neural network architecture which aims at being efficient for short sequences but also at being able to deal with long sequences. IGLOOs core idea is to use the relationships between non-local patches sliced out of the features maps of successively applied convolutions to build a representation for the sequence. We show that the model can deal with dependencies of more than 20,000 steps in a reasonable time frame. We stress test IGLOO on the copy-memory and addition tasks, as well as permuted MNIST (98.4%). For a larger task we apply this new structure to the Wikitext-2 dataset Merity et al. (2017b) and achieve a perplexity in line with baseline Transformers but lower than baseline AWD-LSTM. We also present how IGLOO is already used today in production for bioinformatics tasks.

Motivation & Objective

• Address the challenge of modeling long sequences in neural networks, particularly beyond 20,000 steps.
• Overcome the limitations of RNNs, LSTMs, GRUs, and Transformers in handling very long sequences efficiently.
• Develop a model that maintains efficiency on short sequences while scaling effectively to long sequences.
• Enable practical deployment in real-world applications requiring long-context understanding, such as bioinformatics.

Proposed method

• Slice non-local patches from feature maps generated by successively applied convolutions to extract spatial and temporal relationships.
• Use the relationships between these non-local patches to build a global representation of the sequence.
• Leverage convolutional feature extraction to preserve local patterns while enabling long-range modeling through inter-patch relationships.
• Design the architecture to be computationally efficient for both short and long sequences, avoiding the quadratic complexity of self-attention.
• Integrate the feature slicing mechanism into a feed-forward or hybrid architecture to maintain speed and scalability.
• Train the model end-to-end on sequence modeling benchmarks, using standard objectives like cross-entropy loss.

Experimental results

Research questions

• RQ1 Can a convolution-based architecture effectively model dependencies across sequences longer than 20,000 steps?
• RQ2 How does IGLOO's feature slicing mechanism compare to attention mechanisms in terms of modeling long-range dependencies?
• RQ3 To what extent can IGLOO maintain performance on short sequences while scaling to long sequences?
• RQ4 Can IGLOO achieve competitive results on standard benchmarks like Wikitext-2 and permuted MNIST without relying on attention?
• RQ5 Is IGLOO suitable for real-world production use, particularly in domains like bioinformatics requiring long-context modeling?

Key findings

• IGLOO successfully models sequences with over 20,000 steps in a reasonable time frame, demonstrating scalability beyond typical RNN and Transformer limits.
• On the permuted MNIST task, IGLOO achieves a test accuracy of 98.4%, indicating strong performance on sequence-based image recognition.
• On the Wikitext-2 benchmark, IGLOO achieves a perplexity on par with Transformer baselines, confirming competitive language modeling performance.
• IGLOO achieves a lower perplexity than the AWD-LSTM baseline on Wikitext-2, indicating improved modeling capability.
• IGLOO has already been deployed in production for bioinformatics applications, validating its practical utility and scalability.