[Paper Review] Multinomial Inverse Regression for Text Analysis

Matt Taddy|arXiv (Cornell University)|Dec 9, 2010

Statistical Methods and Inference54 references4 citations

TL;DR

This paper introduces multinomial inverse regression (MNIR) for text analysis, using a gamma-lasso estimation method to perform sentiment-sufficient dimension reduction in high-dimensional text data. By modeling phrase counts as multinomial responses to sentiment annotations, MNIR extracts low-dimensional document scores that preserve sentiment-relevant information, enabling efficient downstream regression with superior performance over existing methods in predictive accuracy and computational efficiency.

ABSTRACT

Text data, including speeches, stories, and other document forms, are often connected to sentiment variables that are of interest for research in marketing, economics, and elsewhere. It is also very high dimensional and difficult to incorporate into statistical analyses. This article introduces a straightforward framework of sentiment-preserving dimension reduction for text data. Multinomial inverse regression is introduced as a general tool for simplifying predictor sets that can be represented as draws from a multinomial distribution, and we show that logistic regression of phrase counts onto document annotations can be used to obtain low dimension document representations that are rich in sentiment information. To facilitate this modeling, a novel estimation technique is developed for multinomial logistic regression with very high-dimension response. In particular, independent Laplace priors with unknown variance are assigned to each regression coefficient, and we detail an efficient routine for maximization of the joint posterior over coefficients and their prior scale. This "gamma-lasso" scheme yields stable and effective estimation for general high-dimension logistic regression, and we argue that it will be superior to current methods in many settings. Guidelines for prior specification are provided, algorithm convergence is detailed, and estimator properties are outlined from the perspective of the literature on non-concave likelihood penalization. Related work on sentiment analysis from statistics, econometrics, and machine learning is surveyed and connected. Finally, the methods are applied in two detailed examples and we provide out-of-sample prediction studies to illustrate their effectiveness.

Motivation & Objective

• To address the challenge of incorporating high-dimensional text data—such as document-level phrase counts—into statistical models when only a small subset is annotated with sentiment variables.
• To develop a dimension reduction framework that preserves sentiment-relevant information by modeling text as a multinomial response to sentiment annotations.
• To propose a novel estimation technique, the gamma-lasso, for high-dimensional multinomial logistic regression with unknown prior variance on coefficients.
• To demonstrate the effectiveness of MNIR through out-of-sample prediction studies across three real-world datasets: U.S. congressional speeches, restaurant reviews, and vote share prediction.
• To compare MNIR with established methods like LDA, Lasso, PLS, and SVM, showing improved predictive accuracy and robustness in sentiment modeling.

Proposed method

• MNIR models the multinomial distribution of phrase counts in documents grouped by sentiment level, with probabilities parameterized via a logistic link function of the sentiment variable.
• The method derives a sufficient reduction score $ z_i = \boldsymbol{\varphi}' \mathbf{f}_i $, where $ \mathbf{f}_i $ is the frequency vector of phrase counts, such that $ y_i \perp\!\!\perp \mathbf{x}_i, m_i \mid z_i $, enabling univariate regression on $ z_i $.
• A gamma-lasso prior is applied to regression coefficients: independent Laplace priors with unknown variance, estimated via a hierarchical Bayesian model with a Gamma hyperprior.
• An efficient optimization routine is developed to maximize the joint posterior over coefficients and their prior scale, using a lower-bound approximation to the log-likelihood and iterative updates for $ \delta $ and $ \varphi $.
• The method uses a convex lower bound on the inverse variance of multinomial probabilities to stabilize estimation, with adaptive updates for $ \delta_{jk}^\star $ based on coefficient changes.
• The framework is implemented in R using the textir package, with comparisons to LDA, Lasso, PLS, SVM, and supervised LDA using standard cross-validation and evaluation metrics.

Experimental results

Research questions

• RQ1 Can multinomial inverse regression effectively reduce high-dimensional text data while preserving sentiment-relevant information for downstream prediction?
• RQ2 How does the gamma-lasso estimation method compare to existing penalized regression and topic modeling techniques in terms of predictive accuracy and computational efficiency?
• RQ3 What is the impact of random effects in MNIR on model performance and run-time in real-world text-sentiment prediction tasks?
• RQ4 How does MNIR perform relative to LDA, PLS, and SVM in out-of-sample prediction on congressional speech, restaurant reviews, and vote share data?
• RQ5 To what extent does the gamma-lasso prior improve estimation stability and variable selection in high-dimensional multinomial logistic regression?

Key findings

• In the vote share prediction task, MNIR with a quadratic forward model achieved an RMSE of 10.7, which was 1.5% better than the best-performing baseline (LDA), and 21% faster than the next fastest method.
• For the two-party vote share prediction, MNIR with a linear forward model achieved an RMSE of 10.7, matching the best-performing method (MNIR with quadratic model), and was 20% faster than the model with random effects.
• In the Republican party classification task, MNIR with logistic forward regression achieved a misclassification rate of 11%, which was 35% better than the next best method (SVM) and 15% better than the baseline.
• For restaurant rating prediction, MNIR with a proportional odds model achieved an RMSE of 1.08, which was 1% better than the best baseline (LDA), and 15% faster than the model with random effects.
• The gamma-lasso method showed stable estimation across all datasets, with the best-performing MNIR model (s = 10⁻²) achieving the lowest RMSE and misclassification rates in all three tasks.
• The MNIR framework demonstrated robustness to model specification, with minimal performance degradation when random effects were excluded, while reducing run-time by 15–40%.