[Paper Review] Transformers for Limit Order Books

James Wallbridge|arXiv (Cornell University)|Feb 29, 2020

Stock Market Forecasting Methods40 references6 citations

TL;DR

This paper proposes TransLOB, a novel deep learning architecture that combines dilated causal convolutional networks with masked multi-head self-attention to predict price movements from limit order book (LOB) data. By leveraging local feature extraction and relational context modeling while preserving causal order, TransLOB achieves state-of-the-art performance on the FI-2010 LOB benchmark, outperforming prior models such as LSTM, CNN-LSTM, and DeepLOB, particularly at longer prediction horizons (e.g., 88.12% accuracy at k=20).

ABSTRACT

We introduce a new deep learning architecture for predicting price movements from limit order books. This architecture uses a causal convolutional network for feature extraction in combination with masked self-attention to update features based on relevant contextual information. This architecture is shown to significantly outperform existing architectures such as those using convolutional networks (CNN) and Long-Short Term Memory (LSTM) establishing a new state-of-the-art benchmark for the FI-2010 dataset.

Motivation & Objective

• To develop a deep learning model that learns relevant features directly from raw limit order book data without relying on hand-crafted features or market assumptions.
• To improve upon existing architectures like CNNs and LSTMs by incorporating both local temporal feature extraction and long-range relational dependencies in time-series LOB data.
• To establish a new state-of-the-art benchmark for price movement prediction on the FI-2010 dataset using a model with strong inductive biases for causality and attention.
• To enhance model interpretability by analyzing attention distributions, offering insights into which historical events the model finds most relevant for predictions.

Proposed method

• The model uses dilated causal convolutional layers to extract local temporal features from raw LOB data, with receptive fields expanded via increasing dilation rates.
• The extracted features are processed through multiple masked multi-head self-attention blocks, where each query attends only to past keys, preserving causal structure.
• The architecture integrates residual connections and layer normalization within each transformer block to stabilize training and improve gradient flow.
• Positional embeddings are used to encode relative temporal positions, enabling attention mechanisms to distinguish between temporal distances.
• The model is trained end-to-end using standard cross-entropy loss for binary classification of price movement direction.
• A causal masking mechanism ensures that no future information is used during attention computation, maintaining temporal consistency.

Experimental results

Research questions

• RQ1 Can a hybrid architecture combining dilated causal convolutions and masked self-attention outperform existing models like LSTMs and CNNs in predicting price movements from LOB data?
• RQ2 How effectively can self-attention mechanisms capture long-range dependencies in high-dimensional, noisy LOB sequences while respecting temporal causality?
• RQ3 To what extent does the model’s attention mechanism highlight meaningful temporal patterns in the LOB data, improving interpretability compared to RNN-based models?
• RQ4 Does the proposed architecture achieve superior sample efficiency and generalization on LOB prediction tasks compared to prior state-of-the-art models?
• RQ5 How does performance scale across different prediction horizons (k=10, 20, 50, 100) on the FI-2010 dataset?

Key findings

• TransLOB achieves 78.78% accuracy at prediction horizon k=10, significantly outperforming the previous SOTA model C(TABL) at 73.74% and DeepLOB at 74.85%.
• At k=20, TransLOB attains 88.12% accuracy, surpassing DeepLOB’s 80.51% and C(TABL)’s 79.87%, demonstrating strong performance at longer horizons.
• At k=50, TransLOB reaches 91.62% accuracy, indicating robustness to extended prediction windows and superior long-range modeling capability.
• At k=100, TransLOB maintains high performance with 91.61% F1-score, confirming its stability and scalability across increasingly distant prediction targets.
• Attention visualization reveals that multiple heads focus on recent LOB events, particularly in heads two and three, aligning with intuitive expectations for price prediction.
• The model exhibits high sample efficiency, suggesting it generalizes well despite the limited size of the FI-2010 dataset, and shows strong potential for scaling to larger datasets with further tuning.