[Paper Review] Decomposing Physician Disagreement in HealthBench

Satya Borgohain, Roy Mariathas|arXiv (Cornell University)|Feb 26, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

The paper analyzes variance in physician disagreement in HealthBench, finding that rubric identity explains a portion of label variance but little of disagreement variance, and that most disagreement is driven by a large case-level residual; reducible uncertainty can double disagreement odds, suggesting evaluation design improvements.

ABSTRACT

We decompose physician disagreement in the HealthBench medical AI evaluation dataset to understand where variance resides and what observable features can explain it. Rubric identity accounts for 15.8% of met/not-met label variance but only 3.6-6.9% of disagreement variance; physician identity accounts for just 2.4%. The dominant 81.8% case-level residual is not reduced by HealthBench's metadata labels (z = -0.22, p = 0.83), normative rubric language (pseudo R^2 = 1.2%), medical specialty (0/300 Tukey pairs significant), surface-feature triage (AUC = 0.58), or embeddings (AUC = 0.485). Disagreement follows an inverted-U with completion quality (AUC = 0.689), confirming physicians agree on clearly good or bad outputs but split on borderline cases. Physician-validated uncertainty categories reveal that reducible uncertainty (missing context, ambiguous phrasing) more than doubles disagreement odds (OR = 2.55, p < 10^(-24)), while irreducible uncertainty (genuine medical ambiguity) has no effect (OR = 1.01, p = 0.90), though even the former explains only ~3% of total variance. The agreement ceiling in medical AI evaluation is thus largely structural, but the reducible/irreducible dissociation suggests that closing information gaps in evaluation scenarios could lower disagreement where inherent clinical ambiguity does not, pointing toward actionable evaluation design improvements.

Motivation & Objective

• Understand where variance in physician disagreement arises in HealthBench.
• Quantify contributions of rubric identity, physician identity, and metadata to disagreement variance.
• Assess whether observable features explain disagreement and identify actionable evaluation-design improvements.

Proposed method

• Decompose variance in HealthBench disagreement across rubric identity, physician identity, and case-level factors.
• Evaluate the predictive power of metadata labels, pseudo R^2 from normative rubric language, medical specialty, surface features, and embeddings (AUCs).
• Model physician-validated uncertainty categories to separate reducible vs irreducible uncertainty and their effects on disagreement odds.

Experimental results

Research questions

• RQ1 What portion of disagreement variance is explained by rubric identity, physician identity, and metadata?
• RQ2 Do observable features (metadata, language, specialty, surface features, embeddings) predict disagreement?
• RQ3 Is disagreement higher for borderline cases and how does completion quality affect agreement?
• RQ4 How do reducible and irreducible uncertainties impact disagreement odds?

Key findings

• Rubric identity accounts for 15.8% of met/not-met label variance but only 3.6-6.9% of disagreement variance.
• Physician identity accounts for 2.4% of disagreement variance.
• A large 81.8% case-level residual remains unexplained by analyzed features.
• Metadata labels (z = -0.22, p = 0.83), pseudo R^2 for rubric language (1.2%), medical specialty (no significant Tukey pairs), surface-feature triage (AUC = 0.58), and embeddings (AUC = 0.485) do not reduce the residual.
• Disagreement follows an inverted-U with completion quality (AUC = 0.689).
• Reducible uncertainty (missing context, ambiguous phrasing) more than doubles disagreement odds (OR = 2.55, p < 10^(-24)); irreducible uncertainty has no effect (OR = 1.01, p = 0.90).
• Even reducible/irreducible uncertainties explain only ~3% of total variance.
• Conclusion: the agreement ceiling is largely structural, and closing information gaps could lower disagreement where clinical ambiguity does not apply.