[Paper Review] HappyMap : A Generalized Multicalibration Method

Anastasios N. Angelopoulos, Travé-Massuyès, Louise+2 more|arXiv (Cornell University)|Jan 1, 2023

Explainable Artificial Intelligence (XAI)Computer Science187 citations

TL;DR

This paper introduces conformal prediction as a distribution-free method for generating statistically valid uncertainty sets in machine learning, applicable to any pre-trained model such as neural networks. It guarantees marginal coverage—e.g., 90% of prediction sets contain the true label—without distributional assumptions, using a simple calibration step with a quantile of conformal scores, enabling reliable uncertainty quantification across diverse tasks like classification, regression, and time-series prediction.

ABSTRACT

Modern complex systems, such as radiotherapy machines, require robust strategies for fault detection, diagnosis, and prognosis to ensure operational continuity and patient safety. While data-driven methods have gained traction, few studies address diagnostic and prognostic tasks using multimodal operational data under unsupervised or semi-supervised learning settings. This gap is particularly critical given the scarcity of labeled failure data in real-world environments. This work aims to design a unified approach for fault detection, diagnosis, and prognosis using multimodal data in the absence of complete labeling. To this end, autoencoders (AEs) are employed due to their suitability for unsupervised and self-supervised learning, flexibility in handling heterogeneous data, and ability to construct latent representations optimized for various downstream tasks. A specific implementation based on a Long Short-Term Memory β-Variational Autoencoder (LSTM-β-VAE) was developed to detect anomalies in machine logs. This framework is applied to TomoTherapy® systems - a highly complex and under-explored use case within the radiotherapy domain. Initial results demonstrate strong anomaly detection performance on both a public benchmark dataset (HDFS) and a proprietary dataset derived from real-world TomoTherapy® machine faults. Beyond methodology, the paper includes a concise literature review of multimodal learning and data-driven diagnosis and prognosis with a focus on AEs. Based on this review, key research directions are identified for the continuation of the thesis, especially the integration of explainable AI as a means to enhance diagnosis capabilities in the absence of labeled faults.

Motivation & Objective

• To provide a self-contained, accessible introduction to conformal prediction for practitioners and researchers.
• To demonstrate how conformal prediction enables valid, distribution-free uncertainty sets for any black-box model.
• To extend conformal prediction to complex scenarios such as distribution shift, time-series, outliers, and structured outputs.
• To promote the adoption of conformal prediction in high-stakes applications through code, examples, and real-world use cases.

Proposed method

• Uses a calibration dataset to compute conformal scores, defined as 1 minus the model's softmax output for the true label.
• Employs the ⌈(n+1)(1−α)⌉/n empirical quantile of these scores to determine a threshold for prediction sets.
• Constructs prediction sets C(Xtest) = {y : ˆf(Xtest)y ≥ 1−q̂}, including all classes with sufficiently high predicted probability.
• Applies the method to classification, regression (via conformalized quantile regression), and scalar uncertainty estimation.
• Extends the framework to group-balanced, class-conditional, and risk-controlled prediction using advanced conformal techniques.
• Supports practical deployment via Jupyter notebooks and a scikit-learn-compatible library (MAPIE), enabling real-world integration.

Experimental results

Research questions

• RQ1 How can conformal prediction be used to generate valid prediction sets for any pre-trained model without distributional assumptions?
• RQ2 What are the key practical considerations for implementing conformal prediction, such as calibration set size and coverage evaluation?
• RQ3 How can conformal prediction be adapted to handle distribution shift, outliers, and time-series data?
• RQ4 Can conformal prediction be extended to structured outputs and risk control in high-stakes applications?
• RQ5 What are the most effective ways to calibrate scalar uncertainty estimates (e.g., standard deviation) using conformal methods?

Key findings

• Conformal prediction guarantees marginal coverage: P(Ytest ∈ C(Xtest)) ∈ [1−α, 1−α + 1/(n+1)], valid non-asymptotically and without model or data distribution assumptions.
• Prediction sets adaptively grow larger for uncertain or hard inputs, providing a natural measure of model confidence.
• The method is robust under distribution shift, with extensions like weighted conformal prediction and adaptive quantile re-estimation for data streams.
• Conformalized quantile regression inherits the uniform convergence properties of quantile regression, ensuring near-conditional coverage asymptotically.
• Real-world applications include election forecasting (The Washington Post), clinical risk prediction, robotic planning, and fairness in criminal justice—demonstrating practical utility.
• The availability of code, Jupyter notebooks, and the MAPIE library enables broad, accessible adoption across research and industry.