[Paper Review] Time-Dynamic Estimates of the Reliability of Deep Semantic Segmentation Networks

Kira Maag, Matthias Rottmann+1 more|arXiv (Cornell University)|Nov 12, 2019

Anomaly Detection Techniques and Applications46 references4 citations

TL;DR

This paper proposes a time-dynamic framework for assessing the reliability of deep semantic segmentation networks by tracking segments across video frames and using temporal metrics to improve uncertainty quantification. By constructing time series of frame-level uncertainty metrics and applying meta-classification and meta-regression models, the method achieves significant improvements—up to 6.78 pp in accuracy and 5.63 pp in R²—over single-frame baselines, particularly with gradient boosting and longer time series.

ABSTRACT

In the semantic segmentation of street scenes with neural networks, the reliability of predictions is of highest interest. The assessment of neural networks by means of uncertainties is a common ansatz to prevent safety issues. As in applications like automated driving, video streams of images are available, we present a time-dynamic approach to investigating uncertainties and assessing the prediction quality of neural networks. We track segments over time and gather aggregated metrics per segment, thus obtaining time series of metrics from which we assess prediction quality. This is done by either classifying between intersection over union equal to 0 and greater than 0 or predicting the intersection over union directly. We study different models for these two tasks and analyze the influence of the time series length on the predictive power of our metrics.

Motivation & Objective

• Address the lack of temporal uncertainty assessment in semantic segmentation, especially for safety-critical applications like automated driving.
• Improve reliability estimation of segmentation predictions by leveraging temporal consistency across video frames.
• Develop a lightweight tracking method to associate predicted segments across consecutive frames for time-series analysis.
• Investigate whether time-series metrics enhance meta-classification (IoU > 0 or not) and meta-regression (direct IoU prediction) performance compared to single-frame methods.
• Evaluate the impact of time series length and model architecture (e.g., gradient boosting, neural networks) on prediction reliability.

Proposed method

• Track predicted segments across video frames using a lightweight, segmentation-specific tracking algorithm to maintain object identity over time.
• Extract frame-level uncertainty metrics (e.g., softmax entropy, highest probability, MC Dropout variance) from the segmentation network’s output for each segment.
• Construct time series of these metrics per segment to capture temporal dynamics and stability.
• Use the time series as input to meta-models for two tasks: (1) meta-classification (predicting whether IoU > 0), and (2) meta-regression (predicting actual IoU values).
• Train and evaluate multiple models—gradient boosting, feedforward neural networks, and linear models—on the time-series inputs.
• Utilize both real annotations and pseudo-labels (from a strong reference model) to train meta-models, especially when ground truth is limited.

Experimental results

Research questions

• RQ1 Can temporal information from video sequences improve the reliability estimation of semantic segmentation predictions compared to single-frame methods?
• RQ2 How does the length of the time series affect the predictive performance of meta-classification and meta-regression models?
• RQ3 To what extent can meta-models trained on pseudo-labels (without ground truth) achieve comparable performance to those trained on real annotations?
• RQ4 Which model architecture (e.g., gradient boosting, neural networks) performs best when using time-series metrics as input?
• RQ5 Does incorporating temporal dynamics reduce overfitting in low-data regimes, particularly when using long time series?

Key findings

• The proposed time-dynamic approach improves meta-classification accuracy by up to 6.78 percentage points and AUROC by 5.04 pp compared to single-frame baselines.
• Meta-regression performance increases by 5.63 pp in R², reaching up to 87.51% (±0.61%) on the KITTI dataset with gradient boosting and time-series inputs.
• Models trained on pseudo-labels alone achieve performance close to those trained on real ground truth, indicating that reliable meta-models can be learned without full annotations.
• Overfitting occurs when time series are too long, especially with gradient boosting, due to limited training data (e.g., only 4,877 segments in the VIPER dataset), limiting gains from longer sequences.
• The best-performing model (gradient boosting with time-series features) achieves an R² of 87.51% and AUROC of 88.68% on the KITTI dataset, outperforming prior single-frame methods.
• Time-series inputs consistently improve performance across all datasets and model types, demonstrating the value of temporal consistency in uncertainty estimation.