[Paper Review] Using Machine Learning and Alternative Data to Predict Movements in Market Risk

Thomas Dierckx, Jesse Davis+1 more|arXiv (Cornell University)|Sep 16, 2020

Stock Market Forecasting Methods4 citations

TL;DR

This paper investigates whether machine learning models can predict daily movements in Apple's market-implied volatility using quantitative market data and alternative data sources like Google News and Wikipedia page views. While only market data yielded the highest predictive accuracy (64.2% balanced accuracy with SVM), the study reveals non-linear relationships between Wikipedia traffic and implied volatility, suggesting potential for improved models with better feature engineering or non-linear algorithms.

ABSTRACT

Using machine learning and alternative data for the prediction of financial markets has been a popular topic in recent years. Many financial variables such as stock price, historical volatility and trade volume have already been through extensive investigation. Remarkably, we found no existing research on the prediction of an asset's market implied volatility within this context. This forward-looking measure gauges the sentiment on the future volatility of an asset, and is deemed one of the most important parameters in the world of derivatives. The ability to predict this statistic may therefore provide a competitive edge to practitioners of market making and asset management alike. Consequently, in this paper we investigate Google News statistics and Wikipedia site traffic as alternative data sources to quantitative market data and consider Logistic Regression, Support Vector Machines and AdaBoost as machine learning models. We show that movements in market implied volatility can indeed be predicted through the help of machine learning techniques. Although the employed alternative data appears to not enhance predictive accuracy, we reveal preliminary evidence of non-linear relationships between features obtained from Wikipedia page traffic and movements in market implied volatility.

Motivation & Objective

• To determine whether machine learning can predict daily direction of movement in market-implied volatility for Apple stock.
• To evaluate the impact of alternative data—specifically Google News counts and Wikipedia page views—on predictive performance.
• To assess whether non-linear relationships between alternative data features and implied volatility exist.
• To explore the feasibility of using limited options data for volatility movement prediction with ML models.
• To identify potential improvements in target construction and model calibration for future research.

Proposed method

• Used a sliding window walk-forward validation with a 379-day training window and one-day out-of-sample prediction, resulting in 106 test cases.
• Extracted features from quantitative market data (e.g., returns, volatility, volume), Google News counts, and Wikipedia page views.
• Applied three machine learning models: Logistic Regression, RBF-kernel SVM, and AdaBoost, all with default scikit-learn configurations.
• Evaluated performance using balanced accuracy due to class imbalance in the target variable (59% down, 41% up movements).
• Conducted ablation studies across five data source scenarios: market data only, market + Google News, market + Wikipedia, market + both, and market + both + additional features.
• Explored model confidence by analyzing predicted probabilities for correlation with precision and movement magnitude.

Experimental results

Research questions

• RQ1 Can machine learning models predict the direction of next-day movement in market-implied volatility for Apple stock?
• RQ2 Does incorporating alternative data (Google News and Wikipedia) improve predictive accuracy compared to quantitative market data alone?
• RQ3 Are there non-linear relationships between Wikipedia traffic features and movements in implied volatility that linear models may miss?
• RQ4 How does class imbalance affect model performance, and does balanced accuracy provide a more reliable evaluation metric?
• RQ5 Can model confidence scores (predicted probabilities) be used to identify more precise predictions or correlate with movement magnitude?

Key findings

• The best predictive performance was achieved using only quantitative market data, with a balanced accuracy of 64.2% using an RBF-kernel SVM.
• The inclusion of Google News counts consistently degraded performance across all models, suggesting limited predictive value for this data source.
• Wikipedia page views showed a non-linear relationship with implied volatility movements, as evidenced by AdaBoost outperforming other models when these features were included.
• Logistic Regression performed poorly with Wikipedia features, indicating that the relationship is likely non-linear and better captured by ensemble methods.
• The best overall result with alternative data was 63.0% balanced accuracy using AdaBoost with market and Wikipedia features, still below the market-only baseline.
• The study found no significant improvement from combining all data sources, suggesting that feature noise or model limitations may hinder generalization on small datasets.