[Paper Review] Effect of the output activation function on the probabilities and errors in medical image segmentation

Lars Nieradzik, Gerik Scheuermann+2 more|arXiv (Cornell University)|Sep 2, 2021

Medical Image Segmentation Techniques46 references4 citations

TL;DR

This paper investigates the impact of different output activation functions on segmentation accuracy and probability calibration in medical image segmentation. It evaluates seven activation functions—sigmoid, CDF, arctangent, hardtanh, inverse square root, linear, and softsign—across three loss functions (BCE, Dice, MSE) on four medical datasets, finding that arctangent outperforms sigmoid under Dice loss and that activation function steepness influences both error rates and probability calibration.

ABSTRACT

The sigmoid activation is the standard output activation function in binary classification and segmentation with neural networks. Still, there exist a variety of other potential output activation functions, which may lead to improved results in medical image segmentation. In this work, we consider how the asymptotic behavior of different output activation and loss functions affects the prediction probabilities and the corresponding segmentation errors. For cross entropy, we show that a faster rate of change of the activation function correlates with better predictions, while a slower rate of change can improve the calibration of probabilities. For dice loss, we found that the arctangent activation function is superior to the sigmoid function. Furthermore, we provide a test space for arbitrary output activation functions in the area of medical image segmentation. We tested seven activation functions in combination with three loss functions on four different medical image segmentation tasks to provide a classification of which function is best suited in this application scenario.

Motivation & Objective

• To identify optimal output activation functions for medical image segmentation beyond the standard sigmoid.
• To analyze how the asymptotic behavior of activation functions affects prediction probabilities and segmentation errors.
• To evaluate the interplay between activation functions and loss functions (BCE, Dice, MSE) in medical segmentation tasks.
• To establish a reproducible test space for comparing arbitrary output activation functions in medical image segmentation.
• To improve both segmentation accuracy and probability calibration in deep learning models for clinical applications.

Proposed method

• Evaluated seven output activation functions: sigmoid, standard normal CDF, arctangent, hardtanh, inverse square root, linear, and softsign.
• Combined each activation with three loss functions: binary cross-entropy (BCE), dice loss, and mean squared error (MSE).
• Trained neural networks on four public medical image segmentation datasets: ACDC, ISLES, Kvasir-SEG, and Medical Segmentation Decathlon.
• Assessed performance using two metrics: negative log-likelihood (for calibration) and dice coefficient (for segmentation accuracy).
• Analyzed the rate of change (derivative) of activation functions to correlate asymptotic behavior with model performance.
• Provided a comprehensive open test space with code and results for reproducible evaluation of new activation functions.

Experimental results

Research questions

• RQ1 How does the rate of change of the output activation function affect segmentation error and probability calibration?
• RQ2 Which output activation function yields the best segmentation performance when combined with binary cross-entropy, dice loss, or mean squared error?
• RQ3 Is the standard sigmoid activation function optimal for medical image segmentation, or are alternatives superior?
• RQ4 How does the choice of loss function interact with the choice of activation function in terms of model performance?
• RQ5 Can alternative statistical activation functions (e.g., probit, linear, CDF) outperform the sigmoid in medical segmentation tasks?

Key findings

• The arctangent activation function outperformed the sigmoid function under dice loss, achieving superior segmentation accuracy.
• Activation functions with a higher rate of change (steeper derivative) produced better segmentation results, particularly in terms of dice coefficient.
• Functions with a slower rate of change, such as the standard normal CDF, led to better-calibrated probabilities, as measured by negative log-likelihood.
• The sigmoid activation function is not optimal for medical image segmentation, as alternative functions like CDF and arctangent show improved performance.
• The choice of loss function significantly influences the optimal activation function, with dice loss favoring arctangent and BCE favoring steeper functions.
• The proposed test space enables systematic evaluation of any output activation function in medical image segmentation, supporting future benchmarking and model development.