[Paper Review] Robust Bayesian Classification Using an Optimistic Score Ratio

Viet Anh Nguyen, Nian Si+1 more|arXiv (Cornell University)|Jul 8, 2020

Advanced Statistical Methods and Models4 citations

TL;DR

This paper proposes a robust Bayesian binary classifier using an optimistic score ratio to handle uncertainty in class-conditional distributions when only limited data is available. By optimizing over an ambiguity set defined by constraints on mean and covariance, the method ensures statistical robustness and computational tractability while achieving strong performance on synthetic and real-world data.

ABSTRACT

We build a Bayesian contextual classification model using an optimistic score ratio for robust binary classification when there is limited information on the class-conditional, or contextual, distribution. The optimistic score searches for the distribution that is most plausible to explain the observed outcomes in the testing sample among all distributions belonging to the contextual ambiguity set which is prescribed using a limited structural constraint on the mean vector and the covariance matrix of the underlying contextual distribution. We show that the Bayesian classifier using the optimistic score ratio is conceptually attractive, delivers solid statistical guarantees and is computationally tractable. We showcase the power of the proposed optimistic score ratio classifier on both synthetic and empirical data.

Motivation & Objective

• To address model uncertainty in class-conditional distributions when training data is limited or heterogeneous.
• To develop a Bayesian classification framework that remains statistically reliable despite misspecification of likelihoods.
• To incorporate structural constraints on mean and covariance into the ambiguity set for robust inference.
• To ensure computational tractability while maintaining strong theoretical guarantees in distributionally robust optimization.
• To provide a decision-theoretic approach that minimizes misclassification risk under ambiguity.

Proposed method

• Uses a Bayesian decision framework where the posterior probability is minimized under model uncertainty.
• Defines an ambiguity set for class-conditional distributions using constraints on mean vector and covariance matrix.
• Applies distributionally robust optimization by maximizing the worst-case likelihood ratio over the ambiguity set.
• Employs an optimistic score ratio that selects the most plausible distribution within the ambiguity set to guide classification.
• Derives a closed-form solution via dual optimization, reducing the problem to a one-dimensional convex minimization over a parameter γ.
• Uses Lagrangian duality and matrix inversion identities to derive the optimal classifier parameters, including mean and covariance adjustments.

Experimental results

Research questions

• RQ1 How can Bayesian classification be made robust when class-conditional distributions are uncertain and only partial information (mean and covariance) is available?
• RQ2 What is the optimal way to define an ambiguity set for class-conditional distributions that balances flexibility and statistical reliability?
• RQ3 Can a computationally tractable classifier be derived that maintains strong theoretical guarantees under distributional uncertainty?
• RQ4 How does the optimistic score ratio improve classification performance compared to standard Bayesian methods under model misspecification?
• RQ5 What is the structure of the optimal classifier when ambiguity is defined via moment constraints on the feature distribution?

Key findings

• The proposed optimistic score ratio classifier is computationally tractable and reduces to a one-dimensional convex optimization problem over a single parameter γ.
• The objective function in the dual problem is strictly convex and tends to infinity as γ approaches infinity, guaranteeing a unique minimizer.
• The optimal solution for the covariance matrix is derived in closed form using a weighted combination of the nominal covariance and the Mahalanobis distance of the test point.
• The method achieves robustness by considering the most favorable distribution within the ambiguity set, leading to improved generalization under covariate shift.
• Empirical results on synthetic and real data demonstrate the classifier's strong performance and stability under model uncertainty.
• Theoretical analysis confirms that the method delivers solid statistical guarantees, including consistency and robustness to distributional shifts.