[Paper Review] Model Prior Distribution for Variable Selection in Linear Regression Models

J Lee, Lee, Jeong Eun|arXiv (Cornell University)|Dec 26, 2015

Statistical Methods and Bayesian Inference30 references3 citations

TL;DR

This paper proposes a novel objective model prior distribution for Bayesian variable selection in linear regression, based on Kullback–Leibler divergence losses to quantify model worth. It outperforms uniform and Scott & Berger priors in simulation studies under non-informative settings, particularly in accurately estimating model size and improving posterior frequentist performance.

ABSTRACT

In this work we discuss a novel model prior probability for variable selection in linear regression. The idea is to determine the prior mass in an objective sense, by considering the worth of each of the possible regression models, given the number of covariates under consideration. Through a simulation study, we show that the proposed prior outperforms the uniform prior and the Scott \& Berger prior in a scenario of no prior knowledge about the size of the true regression models. We illustrate the use of the prior using two well-known data sets with, respectively, 15 and 4 covariates.

Motivation & Objective

• To develop an objective, data-driven model prior distribution for variable selection in linear regression that reflects the inherent worth of each model.
• To address the limitations of uniform and Scott & Berger priors in scenarios with no prior knowledge about the true model size.
• To improve the frequentist performance of posterior model size estimation through a principled loss-based approach.
• To provide a minimally informative yet statistically coherent alternative to existing model priors in high-dimensional variable selection.

Proposed method

• The proposed prior assigns model probabilities based on the Kullback–Leibler (KL) divergence between competing models, measuring their statistical distinguishability.
• Model worth is quantified by minimizing the KL divergence between a model and its alternatives, with lower divergence indicating higher model worth.
• The prior is constructed to depend on model size (number of covariates), ensuring it adapts to the complexity of the model space.
• The method uses the expected KL divergence under the sampling distribution to derive a prior that penalizes models with poor predictive fidelity.
• The prior is compared against uniform and Scott & Berger priors via simulation and real data analysis, focusing on posterior model size performance.
• The approach integrates into the Bayesian framework by updating prior uncertainty about the true model with observed data to yield a posterior distribution.

Experimental results

Research questions

• RQ1 How can an objective model prior be constructed that reflects the relative worth of regression models without relying on subjective assumptions?
• RQ2 Does a loss-based model prior improve frequentist performance in model size estimation compared to uniform and Scott & Berger priors?
• RQ3 Can the proposed prior better identify the true model when no prior information is available about its size?
• RQ4 How does the proposed prior perform on real-world data sets with known structure, such as the US crime and Hald data?
• RQ5 What is the impact of model complexity (number of covariates) on the performance of the proposed prior?

Key findings

• The proposed loss-based model prior significantly outperforms the uniform prior in terms of frequentist accuracy of posterior model size estimation in simulation studies.
• The prior demonstrates superior performance compared to the Scott & Berger prior, especially in scenarios with no prior knowledge about the true model size.
• In simulations, the posterior distribution of model size under the proposed prior is more concentrated around the true model size than under the other priors.
• On the US crime data set (15 covariates), the proposed prior identified a more parsimonious and stable model compared to the benchmark priors.
• On the Hald data set (4 covariates), the proposed prior produced a posterior model distribution that better aligned with known regression structure and improved predictive performance.
• The method's dependence on model size ensures it naturally penalizes overly complex models while maintaining sensitivity to true signals.