[Paper Review] BinaryPPO: Efficient Policy Optimization for Binary Classification

Punya Syon Pandey, Zhijing Jin|arXiv (Cornell University)|Feb 2, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

BinaryPPO reframes binary classification as an offline reinforcement learning problem using a confidence-weighted reward, achieving large accuracy gains over supervised baselines across eight domain benchmarks.

ABSTRACT

Supervised fine-tuning (SFT) is the standard approach for binary classification tasks such as toxicity detection, factuality verification, and causal inference. However, SFT often performs poorly in real-world settings with label noise, class imbalance, or sparse supervision. We introduce BinaryPPO, an offline reinforcement learning large language model (LLM) framework that reformulates binary classification as a reward maximization problem. Our method leverages a variant of Proximal Policy Optimization (PPO) with a confidence-weighted reward function that penalizes uncertain or incorrect predictions, enabling the model to learn robust decision policies from static datasets without online interaction. Across eight domain-specific benchmarks and multiple models with differing architectures, BinaryPPO improves accuracy by 40-60 percentage points, reaching up to 99%, substantially outperforming supervised baselines. We provide an in-depth analysis of the role of reward shaping, advantage scaling, and policy stability in enabling this improvement. Overall, we demonstrate that confidence-based reward design provides a robust alternative to SFT for binary classification. Our code is available at https://github.com/psyonp/BinaryPPO.

Motivation & Objective

• Motivation: supervised fine-tuning struggles with label noise, class imbalance, and sparse supervision in binary tasks.
• Goal: learn robust binary decision policies by maximizing a reward signal that encodes model confidence and correctness.
• Scope: evaluate on eight domain benchmarks spanning toxicity detection, factuality, and causal reasoning, across multiple model architectures.
• Claim: confidence-weighted reward design enables robust, offline policy optimization without online interaction.

Proposed method

• Reformulate binary classification as a decision problem under uncertainty optimized with a variant of PPO.
• Define a probabilistic reward r(x,a,y) that scales with model confidence via f(π_old(a|x)) and a correctness signal s(a,y).
• Use a value network V_φ(x) to compute advantage A(x,a,y)=r(x,a,y)−V_φ(x).
• Optimize a combined objective that includes the PPO loss, value loss, supervised cross-entropy regularization, and entropy regularization to encourage exploration.
• Incorporate entropy regularization and an equal data sampling ablation to study stability and generalization.
• Provide an offline training framework applicable to existing datasets without online feedback.

Experimental results

Research questions

• RQ1 Can confidence-weighted, offline reinforcement learning improve binary classification accuracy over standard supervised fine-tuning and PPO baselines?
• RQ2 How do reward shaping, advantage scaling, and policy stability impact learning and generalization across domains?
• RQ3 Does BinaryPPO preserve normative safety and fairness properties while improving task accuracy?
• RQ4 Is the method robust to distribution shifts and cross-domain generalization (OOD scenarios) across model architectures?

Key findings

• BinaryPPO yields large accuracy improvements (up to tens of percentage points) over supervised baselines across eight benchmarks.
• On Qwen-2.5-3B and Gemma-2-2B, BinaryPPO achieves near-saturated accuracies (around 98–99%) on multiple tasks.
• Entropy regularization is essential for stability; removing it leads to dramatic performance collapse, while equal data sampling improves but does not match full BinaryPPO.
• BinaryPPO demonstrates out-of-distribution generalization in causality and moderation tasks, indicating transferable decision boundaries across models.
• Normative evaluations show BinaryPPO largely preserves toxicity resistance and truthfulness, with limited dataset-specific regressions.
• Training dynamics show policy entropy and KL divergence stabilize early, indicating stable convergence.