[Paper Review] Statistical hypothesis testing versus machine-learning binary classification: distinctions and guidelines

Jingyi Jessica Li, Xin Tong|arXiv (Cornell University)|Jul 3, 2020

Gene expression and cancer classificationBiochemistry, Genetics and Molecular Biology68 references48 citations

TL;DR

This paper clarifies the fundamental distinctions between statistical hypothesis testing and machine-learning binary classification, emphasizing their differing goals—inferring population-level truths versus predicting instance-level labels. It proposes five practical guidelines to help data analysts choose the appropriate method based on their analytical objectives, demonstrated through cancer driver gene prediction, where binary classification outperforms traditional hypothesis testing in discovery accuracy.

ABSTRACT

Making binary decisions is a common data analytical task in scientific research and industrial applications. In data sciences, there are two related but distinct strategies: hypothesis testing and binary classification. In practice, how to choose between these two strategies can be unclear and rather confusing. Here we summarize key distinctions between these two strategies in three aspects and list five practical guidelines for data analysts to choose the appropriate strategy for specific analysis needs. We demonstrate the use of those guidelines in a cancer driver gene prediction example.

Motivation & Objective

• To clarify the conceptual and methodological distinctions between statistical hypothesis testing and machine-learning binary classification.
• To address the widespread confusion among practitioners about when to use each approach despite their overlapping applications.
• To provide five practical, actionable guidelines for data analysts to select the appropriate strategy based on their analytical goals.
• To demonstrate the real-world utility of these guidelines through a case study in cancer driver gene prediction.

Proposed method

• The authors compare hypothesis testing and binary classification across three core dimensions: data relationship to decisions, decision rule construction, and evaluation criteria.
• They define four key concepts—binary questions, binary answers, decision rules, and binary decisions—within each framework to clarify their distinct roles.
• The paper uses a cancer driver gene prediction task to empirically compare both strategies, applying cross-validation and AUPRC to evaluate binary classification performance.
• It highlights that hypothesis testing evaluates population-level properties (e.g., differential gene expression), while binary classification predicts instance-level labels (e.g., whether a gene is a driver) from features.
• The authors advocate for using binary classification when the goal is prediction on new instances, especially when training data are representative.
• They illustrate that logistic regression can serve dual roles—both as a classifier and as a hypothesis testing tool—showcasing methodological overlap.

Experimental results

Research questions

• RQ1 What are the key conceptual and methodological differences between statistical hypothesis testing and machine-learning binary classification?
• RQ2 When should a data analyst choose hypothesis testing over binary classification, or vice versa, in real-world applications?
• RQ3 How do the evaluation criteria and decision rule construction processes differ between the two strategies?
• RQ4 Can insights from hypothesis testing improve the design of binary classifiers, and vice versa?
• RQ5 In what scenarios might a single algorithm, such as logistic regression, serve both purposes simultaneously?

Key findings

• Hypothesis testing evaluates whether a population-level property (e.g., differential expression) is statistically significant, while binary classification predicts the label of individual instances (e.g., driver gene status) based on features.
• Binary classification, particularly when evaluated via AUPRC, can lead to more accurate cancer driver gene discoveries than traditional hypothesis testing in the same dataset.
• The study demonstrates that a binary classification approach using mutational signatures achieves higher precision and better discovery performance than prior methods.
• Decision rules in hypothesis testing are derived from test statistics and significance thresholds, whereas in binary classification, they are learned from training data using algorithms like logistic regression or SVMs.
• Despite their differences, the two strategies can inform each other—e.g., test statistics can inspire classifier design, and classifier-extracted features can become targets for hypothesis testing.
• Logistic regression exemplifies a method that serves both as a classification algorithm and a hypothesis testing tool, particularly through the Wald test on individual coefficients.