[Paper Review] Analysis of Video Feature Learning in Two-Stream CNNs on the Example of Zebrafish Swim Bout Classification

Bennet Breier, Arno Onken|arXiv (Cornell University)|Dec 20, 2019

Explainable Artificial Intelligence (XAI)49 references4 citations

TL;DR

This study applies two-stream 3D CNNs with Deep Taylor Decomposition for explainable video feature learning in zebrafish swim bout classification, achieving 96.32% accuracy—6.12% higher than the prior SVM-based state-of-the-art—by identifying tail trunk steadiness as the key discriminative feature, while uncovering and correcting a 'Clever Hans' spurious correlation from agarose artifacts.

ABSTRACT

Semmelhack et al. (2014) have achieved high classification accuracy in distinguishing swim bouts of zebrafish using a Support Vector Machine (SVM). Convolutional Neural Networks (CNNs) have reached superior performance in various image recognition tasks over SVMs, but these powerful networks remain a black box. Reaching better transparency helps to build trust in their classifications and makes learned features interpretable to experts. Using a recently developed technique called Deep Taylor Decomposition, we generated heatmaps to highlight input regions of high relevance for predictions. We find that our CNN makes predictions by analyzing the steadiness of the tail's trunk, which markedly differs from the manually extracted features used by Semmelhack et al. (2014). We further uncovered that the network paid attention to experimental artifacts. Removing these artifacts ensured the validity of predictions. After correction, our best CNN beats the SVM by 6.12%, achieving a classification accuracy of 96.32%. Our work thus demonstrates the utility of AI explainability for CNNs.

Motivation & Objective

• To improve classification accuracy in zebrafish swim bout detection beyond the prior SVM-based state-of-the-art.
• To investigate whether deep learning models learn interpretable, biologically relevant features in video-based behavioral analysis.
• To identify and correct spurious correlations (e.g., from experimental artifacts) that undermine model robustness and interpretability.
• To demonstrate the utility of AI explainability techniques in neuroscience and behavioral biology applications.
• To validate that CNNs can learn more discriminative features than manually engineered ones in zebrafish locomotion analysis.

Proposed method

• Trained a two-stream 3D CNN using optical flow inputs to capture motion dynamics in zebrafish video sequences.
• Applied Deep Taylor Decomposition via the iNNvestigate toolbox to generate saliency heatmaps highlighting input regions most relevant to predictions.
• Used transfer learning with ImageNet-pretrained weights to initialize the spatial and temporal streams.
• Performed data augmentation and normalization on optical flow sequences to improve training stability.
• Identified and removed spurious correlations linked to agarose artifacts in the experimental setup.
• Retrained the model after artifact removal to evaluate performance on corrected data.

Experimental results

Research questions

• RQ1 What features do two-stream CNNs learn for zebrafish swim bout classification, and how do they compare to manually engineered features?
• RQ2 To what extent do experimental artifacts like agarose create spurious correlations in video-based behavioral classification?
• RQ3 Can AI explainability techniques like Deep Taylor Decomposition reveal biologically meaningful patterns in learned features?
• RQ4 Does removing spurious correlations improve model generalization and performance in zebrafish behavior classification?
• RQ5 Can deep learning models outperform traditional SVMs with handcrafted features in this domain, and if so, why?

Key findings

• The CNN achieved a test accuracy of 96.32%, which is 6.12 percentage points higher than the SVM-based method by Semmelhack et al. (2014).
• The model primarily relies on the steadiness of the tail’s trunk to classify swim bouts, with high relevance localized to the trunk edges.
• Heatmaps revealed that the network detects prey bouts via sharp, localized relevance profiles and spontaneous bouts via diffuse, oscillatory patterns.
• A 'Clever Hans' correlation was uncovered: agarose artifacts in the top-left corner moved during spontaneous bouts but not prey bouts, misleading the model.
• After removing the agarose artifact correlation and retraining, the model's performance improved and remained robust, confirming the validity of the corrected features.
• The CNN learned features fundamentally different from those used in Semmelhack et al. (2014), relying less on tail tip dynamics and more on trunk kinematics, which aligns with known behavioral neurobiology.