[Paper Review] Multi-Modal Deep Learning for Credit Rating Prediction Using Text and Numerical Data Streams

Mahsa Tavakoli, Rohitash Chandra+2 more|arXiv (Cornell University)|Apr 21, 2023

Financial Distress and Bankruptcy PredictionBusiness, Management and Accounting3 citations

TL;DR

This paper proposes a multi-modal deep learning framework that fuses text from earnings call transcripts and numerical financial data to predict corporate credit ratings. Using CNN-based models with early-intermediate hybrid fusion and concatenation, it achieves superior performance over complex attention-based models, with text modality proving most predictive—especially during the COVID-19 crisis—while Moody’s ratings show stronger long-term alignment with model predictions.

ABSTRACT

Knowing which factors are significant in credit rating assignment leads to better decision-making. However, the focus of the literature thus far has been mostly on structured data, and fewer studies have addressed unstructured or multi-modal datasets. In this paper, we present an analysis of the most effective architectures for the fusion of deep learning models for the prediction of company credit rating classes, by using structured and unstructured datasets of different types. In these models, we tested different combinations of fusion strategies with different deep learning models, including CNN, LSTM, GRU, and BERT. We studied data fusion strategies in terms of level (including early and intermediate fusion) and techniques (including concatenation and cross-attention). Our results show that a CNN-based multi-modal model with two fusion strategies outperformed other multi-modal techniques. In addition, by comparing simple architectures with more complex ones, we found that more sophisticated deep learning models do not necessarily produce the highest performance; however, if attention-based models are producing the best results, cross-attention is necessary as a fusion strategy. Finally, our comparison of rating agencies on short-, medium-, and long-term performance shows that Moody's credit ratings outperform those of other agencies like Standard & Poor's and Fitch Ratings.

Motivation & Objective

• To evaluate the effectiveness of multi-modal deep learning models in predicting corporate credit ratings using both structured (numerical) and unstructured (textual) data.
• To investigate how different fusion strategies—by level (early, intermediate) and technique (concatenation, cross-attention)—affect model performance.
• To determine whether more complex architectures (e.g., attention-based models) yield better results than simpler ones in credit rating prediction.
• To assess the relative contribution of text and numerical modalities to predictive performance, particularly during periods of market stress like the COVID-19 pandemic.
• To compare the predictive alignment of models with ratings from major agencies (Moody’s, S&P, Fitch), analyzing performance across short-, medium-, and long-term horizons.

Proposed method

• Employs deep learning models including CNN, LSTM, GRU, and BERT to process numerical and textual data streams separately.
• Applies early and intermediate fusion strategies to combine representations from different modalities at different stages of the network.
• Uses concatenation and cross-attention as fusion techniques to integrate features from text and numerical streams.
• Trains and evaluates 16 distinct model configurations combining different backbone networks and fusion strategies on a dataset of corporate earnings call transcripts and financial statements.
• Implements out-of-distribution (OOU) robustness testing to evaluate generalization on unseen data and crisis periods.
• Conducts ablation studies to measure modality-specific contributions and assesses model stability across short-, medium-, and long-term prediction lags.

Experimental results

Research questions

• RQ1 Which multi-modal fusion strategy (early, intermediate, or hybrid) combined with which deep learning architecture yields the highest accuracy in credit rating prediction?
• RQ2 Does the inclusion of textual data from earnings calls significantly improve prediction performance compared to numerical data alone?
• RQ3 Do more complex models like attention-based transformers (e.g., BERT with cross-attention) outperform simpler models like CNNs in this multi-modal setting?
• RQ4 How does model performance vary across short-, medium-, and long-term prediction horizons, and which rating agency’s ratings are most aligned with model predictions?
• RQ5 To what extent does the model maintain predictive power during periods of market stress, such as the COVID-19 pandemic?

Key findings

• A CNN-based model with a hybrid fusion strategy (early and intermediate fusion) and concatenation as the fusion technique achieved the highest performance, outperforming more complex attention-based models.
• Text modality contributed most significantly to prediction accuracy, particularly during the COVID-19 crisis, indicating that management commentary carries critical risk information.
• Despite their complexity, attention-based models did not outperform simpler CNNs unless paired with cross-attention fusion, which was essential for their superior performance.
• The model demonstrated robustness through out-of-distribution (OOU) testing, maintaining predictive power on unseen data and during periods of market volatility.
• Moody’s credit ratings showed stronger alignment with model predictions than those from S&P and Fitch, especially in the medium-term horizon, though differences were relatively small.
• The study confirms that integrating textual and numerical data via multi-modal deep learning enhances predictive accuracy beyond traditional models relying on structured data alone.