[Paper Review] Stacked dense optical flows and dropout layers to predict sperm motility and morphology

Vajira Thambawita, Pål Halvorsen+3 more|arXiv (Cornell University)|Nov 8, 2019

Sperm and Testicular Function19 references4 citations

TL;DR

This paper proposes a deep learning approach using stacked dense optical flows and dropout-regularized multi-layer perceptrons (MLPs) with ResNet-34 to predict sperm motility and morphology from video frames. By combining raw frames with multi-scale optical flow inputs and adding an MLP with dropout to prevent overfitting, the method achieves state-of-the-art mean absolute error (MAE) of 8.825 for motility and 5.293 for morphology.

ABSTRACT

In this paper, we analyse two deep learning methods to predict sperm motility and sperm morphology from sperm videos. We use two different inputs: stacked pure frames of videos and dense optical flows of video frames. To solve this regression task of predicting motility and morphology, stacked dense optical flows and extracted original frames from sperm videos were used with the modified state of the art convolution neural networks. For modifications of the selected models, we have introduced an additional multi-layer perceptron to overcome the problem of over-fitting. The method which had an additional multi-layer perceptron with dropout layers, shows the best results when the inputs consist of both dense optical flows and an original frame of videos.

Motivation & Objective

• To develop a deep learning model that predicts sperm motility and morphology from video sequences with high accuracy.
• To investigate the effectiveness of dense optical flow representations in capturing temporal motion patterns in sperm video data.
• To reduce overfitting in small-scale medical video datasets through architectural modifications like dropout and additional MLP layers.
• To compare the performance of raw frame stacking versus optical flow-enhanced input features in regression-based sperm analysis.
• To establish a baseline for automated computer-aided sperm analysis using end-to-end deep learning on the VISEM dataset.

Proposed method

• Preprocessed sperm videos by extracting nine consecutive grayscale frames (D1) and combining them with RGB frames and two types of dense optical flow images (stride 1 and stride 10) to form a nine-channel input tensor (D2).
• Utilized pre-trained ResNet-34 as the backbone network, fine-tuned for 9-channel input and adapted to output three regression values for motility or morphology.
• Introduced a modified model (M2) with an additional multi-layer perceptron (MLP) after the ResNet-34 features, incorporating dropout layers to mitigate overfitting.
• Trained models using the Adam optimizer with a fixed learning rate of 0.001 and mean squared error (MSE) as the loss function during backpropagation.
• Evaluated models using three-fold cross-validation on the VISEM dataset, with mean absolute error (MAE) used as the primary metric for performance comparison.
• Generated dense optical flows using OpenCV’s Farnebäck algorithm on consecutive frames with different temporal strides to capture motion dynamics at varying scales.

Experimental results

Research questions

• RQ1 How do dense optical flow representations improve the prediction of sperm motility and morphology compared to raw frame stacking?
• RQ2 What is the impact of adding a multi-layer perceptron with dropout on model generalization and overfitting in small medical video datasets?
• RQ3 Which input configuration—raw frames only or raw frames combined with multi-scale optical flows—yields superior performance in sperm quality regression?
• RQ4 Can a modified ResNet-34 architecture with architectural regularization outperform the standard ResNet-34 in this medical video regression task?
• RQ5 What is the optimal combination of input features and model architecture for minimizing mean absolute error in sperm motility and morphology prediction?

Key findings

• The method using both raw frames and dense optical flows (D2 input) with the modified ResNet-34 model (M2) achieved the lowest mean absolute error (MAE) of 8.825 for sperm motility and 5.293 for morphology.
• The M2 model with dropout-regularized MLP consistently outperformed the base M1 model across all input types and folds, demonstrating the effectiveness of architectural regularization.
• The D2 input (combined raw and optical flow features) yielded better results than D1 (stacked raw frames only), indicating that motion information enhances predictive performance.
• The best-performing configuration (D2-M2) showed a 10.5% improvement in motility MAE and 5.3% improvement in morphology MAE compared to the base model with D1 input.
• The model with D2 input and M2 architecture achieved the lowest average MAE across all folds, with fold-specific values of 8.612 (fold 1), 7.873 (fold 2), and 9.991 (fold 3) for motility.
• The use of multi-scale optical flows (stride 1 and stride 10) provided complementary motion cues, contributing to improved temporal modeling and better generalization.