[Paper Review] Bayesian Masking: Sparse Bayesian Estimation with Weaker Shrinkage Bias

Yohei Kondo, Kohei Hayashi+1 more|arXiv (Cornell University)|Sep 3, 2015

Sparse and Compressive Sensing Techniques9 references3 citations

TL;DR

This paper proposes Bayesian Masking (BM), a sparse Bayesian estimation method that avoids regularization on weights by introducing binary latent variables to randomly mask features. By estimating masking rates via variational Bayesian inference on the factorized information criterion (FIC), BM achieves superior sparsity-shrinkage trade-off compared to Lasso and ARD, with accelerated convergence via gradient ascent and reparametrization.

ABSTRACT

A common strategy for sparse linear regression is to introduce regularization, which eliminates irrelevant features by letting the corresponding weights be zeros. However, regularization often shrinks the estimator for relevant features, which leads to incorrect feature selection. Motivated by the above-mentioned issue, we propose Bayesian masking (BM), a sparse estimation method which imposes no regularization on the weights. The key concept of BM is to introduce binary latent variables that randomly mask features. Estimating the masking rates determines the relevance of the features automatically. We derive a variational Bayesian inference algorithm that maximizes the lower bound of the factorized information criterion (FIC), which is a recently developed asymptotic criterion for evaluating the marginal log-likelihood. In addition, we propose reparametrization to accelerate the convergence of the derived algorithm. Finally, we show that BM outperforms Lasso and automatic relevance determination (ARD) in terms of the sparsity-shrinkage trade-off.

Motivation & Objective

• Address the inherent shrinkage bias in traditional sparse regularization methods like Lasso and ARD, which distort relevant feature weights.
• Overcome the limitation of regularization-based sparsity by eliminating direct penalization on weight parameters.
• Develop a novel sparse estimation framework that identifies feature relevance through learnable masking rates rather than weight shrinkage.
• Improve convergence speed of the inference algorithm using gradient ascent and reparametrization techniques grounded in information geometry.
• Demonstrate empirically that BM outperforms Lasso and ARD in balancing sparsity and shrinkage, especially in high-dimensional settings.

Proposed method

• Introduce binary latent variables to stochastically mask features during training, where each feature's masking rate is modeled as a learnable prior.
• Formulate the BM model as a hierarchical Bayesian linear regression with masking probabilities shared across samples but learned per feature.
• Derive a variational Bayesian inference algorithm that maximizes a lower bound of the factorized information criterion (FIC), enabling asymptotically accurate marginal likelihood approximation.
• Implement an EM-like coordinate ascent algorithm with gradient updates to accelerate convergence, leveraging reparametrization for stable optimization.
• Use the FAB (Fisher-Ascent-Blending) algorithm framework to jointly optimize masking rates and model parameters, blending EM and gradient ascent steps.
• Apply reparametrization to the masking rate parameters to improve the stability and speed of gradient-based updates, inspired by information geometry and convergence analysis.

Experimental results

Research questions

• RQ1 Can a sparse estimation method avoid shrinkage bias in relevant feature weights without relying on regularization of the weight parameters?
• RQ2 How does the performance of a masking-based approach compare to traditional regularization methods like Lasso and ARD in terms of sparsity and shrinkage trade-off?
• RQ3 Can variational Bayesian inference based on the factorized information criterion (FIC) effectively estimate feature relevance in a regularization-free framework?
• RQ4 To what extent does combining gradient ascent with reparametrization improve convergence speed in the inference algorithm for the BM model?
• RQ5 Does the proposed method maintain high precision while achieving high recall in feature selection, especially as the number of features increases?

Key findings

• BM achieves the highest F1 score across all tested feature counts (K = 10, 30, 50, 100), indicating the best overall balance between precision and recall in feature selection.
• For K = 50, the hybrid FAB-EM-EG algorithm converged significantly faster than FAB-EM, with a higher number of correctly pruned features per unit time, without increasing error rates.
• The number of wrongly pruned features was nearly identical between FAB-EM-EG and FAB-EM (2.0 ± 1.3 vs. 1.8 ± 1.3), confirming that faster convergence was not due to over-pruning.
• BM achieved the highest recall (up to 0.92) across all K values, indicating superior ability to detect truly irrelevant features compared to Lasso and ARD.
• The analytic forms of the estimators show that BM avoids shrinkage bias by decoupling relevance detection (via masking rates) from weight estimation.
• The combination of gradient ascent and reparametrization led to a sharp acceleration in convergence, as shown in learning trajectories on the β₁–π₁ plane.