[Paper Review] MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection

Kemal Öksüz, Selim Kuzucu+2 more|arXiv (Cornell University)|Sep 26, 2023

COVID-19 diagnosis using AIMedicine3 citations

TL;DR

This paper proposes MoCaE (Mixture of Calibrated Experts), a method that improves object detection by calibrating individual expert detectors before combining their predictions via a refined NMS strategy. By correcting confidence miscalibration—where high-confidence predictions do not reflect actual accuracy—MoCaE achieves state-of-the-art performance, improving AP by up to 2.5 on COCO and reaching 65.1 AP on COCO test-dev and 82.62 AP50 on DOTA.

ABSTRACT

Combining the strengths of many existing predictors to obtain a Mixture of Experts which is superior to its individual components is an effective way to improve the performance without having to develop new architectures or train a model from scratch. However, surprisingly, we find that naïvely combining expert object detectors in a similar way to Deep Ensembles, can often lead to degraded performance. We identify that the primary cause of this issue is that the predictions of the experts do not match their performance, a term referred to as miscalibration. Consequently, the most confident detector dominates the final predictions, preventing the mixture from leveraging all the predictions from the experts appropriately. To address this, when constructing the Mixture of Experts, we propose to combine their predictions in a manner which reflects the individual performance of the experts; an objective we achieve by first calibrating the predictions before filtering and refining them. We term this approach the Mixture of Calibrated Experts and demonstrate its effectiveness through extensive experiments on 5 different detection tasks using a variety of detectors, showing that it: (i) improves object detectors on COCO and instance segmentation methods on LVIS by up to $\sim 2.5$ AP; (ii) reaches state-of-the-art on COCO test-dev with $65.1$ AP and on DOTA with $82.62$ $\mathrm{AP_{50}}$; (iii) outperforms single models consistently on recent detection tasks such as Open Vocabulary Object Detection.

Motivation & Objective

• To address the degradation in performance when naively combining object detectors in a Mixture of Experts (MoE), despite their individual strengths.
• To identify that miscalibration—where predicted confidence does not match actual accuracy—is the primary cause of performance drop in MoEs.
• To propose a method that calibrates expert predictions before ensemble combination, ensuring each expert's true performance is reflected in the final output.
• To demonstrate that calibrated ensembles outperform both individual models and standard MoEs across diverse detection benchmarks, including open-vocabulary detection.

Proposed method

• The method first applies confidence calibration to individual object detectors using temperature scaling and label smoothing to align predicted confidence with actual accuracy.
• It constructs a Mixture of Experts (MoE) by combining predictions from multiple calibrated detectors using a refined Non-Maximum Suppression (NMS) strategy.
• The refinement step prioritizes high-quality detections while suppressing false positives, with attention to IoU and confidence scores post-calibration.
• Calibration is applied independently per expert using a validation set, requiring only a few learnable parameters per detector.
• The final ensemble prediction is formed by merging detections from all calibrated experts, with class-specific NMS applied to avoid duplicate predictions.
• The approach is plug-and-play, compatible with off-the-shelf detectors and requires minimal architectural changes.

Experimental results

Research questions

• RQ1 Why does naive Mixture of Experts (MoE) fail to improve performance when combining multiple object detectors, despite their individual strengths?
• RQ2 To what extent does prediction miscalibration—where confidence scores do not reflect actual accuracy—harm MoE performance in object detection?
• RQ3 Can calibrating individual experts before ensemble combination lead to consistent performance gains across diverse detection tasks and datasets?
• RQ4 How does MoCaE compare to standard Deep Ensembles and single-model baselines on challenging benchmarks like COCO, LVIS, and DOTA?
• RQ5 Does MoCaE maintain its advantage in recent, complex detection tasks such as Open Vocabulary Object Detection?

Key findings

• MoCaE improves object detection performance on COCO by up to 2.5 AP, with a new state-of-the-art result of 65.1 AP on COCO test-dev.
• On the DOTA dataset for rotated object detection, MoCaE achieves 82.62 AP50, setting a new SOTA benchmark.
• On the LVIS instance segmentation benchmark, MoCaE outperforms all single detectors and standard MoEs, achieving 29.4 AP with a 2.3-point gain over the best single model.
• In Open Vocabulary Object Detection, MoCaE consistently outperforms single models and standard MoEs, demonstrating robustness to out-of-distribution classes.
• The method achieves a 1.5 AP gain over single detectors in the most challenging setting (YOLOv7, QueryInst, DyHead), with gains of +2.4 AP on COCO test-dev.
• Ablation studies confirm that calibration is essential: uncalibrated MoEs show no improvement over single models, while calibrated MoEs yield significant gains.