[Paper Review] Evaluation of deep lift pose models for 3D rodent pose estimation based on geometrically triangulated data

Indrani Sarkar, Indranil Maji+4 more|arXiv (Cornell University)|Jun 24, 2021

Zebrafish Biomedical Research Applications14 references4 citations

TL;DR

This paper proposes a robust 3D rodent pose estimation method using deep lift models trained on geometrically triangulated 2D keypoint data from multi-camera setups. It demonstrates that temporal convolutional networks outperform linear residual networks in lifting single-view 2D poses to 3D, achieving a test loss of 0.45±0.003 at an optimal 2.7s temporal window, enabling reliable 3D pose inference in complex behavioral experiments.

ABSTRACT

The assessment of laboratory animal behavior is of central interest in modern neuroscience research. Behavior is typically studied in terms of pose changes, which are ideally captured in three dimensions. This requires triangulation over a multi-camera system which view the animal from different angles. However, this is challenging in realistic laboratory setups due to occlusions and other technical constrains. Here we propose the usage of lift-pose models that allow for robust 3D pose estimation of freely moving rodents from a single view camera view. To obtain high-quality training data for the pose-lifting, we first perform geometric calibration in a camera setup involving bottom as well as side views of the behaving animal. We then evaluate the performance of two previously proposed model architectures under given inference perspectives and conclude that reliable 3D pose inference can be obtained using temporal convolutions. With this work we would like to contribute to a more robust and diverse behavior tracking of freely moving rodents for a wide range of experiments and setups in the neuroscience community.

Motivation & Objective

• To enable robust 3D pose estimation of freely moving rodents in complex laboratory environments where multi-camera triangulation is hindered by occlusions and technical constraints.
• To develop a reliable geometric calibration procedure using orthogonal bottom and side-view cameras to generate high-quality 3D training data from 2D keypoint detections.
• To evaluate the performance of two deep lift pose models—linear residual network and dilated temporal convolutional network—on rodent pose lifting from single-view inputs.
• To identify optimal inference perspectives and temporal window settings for accurate 3D pose reconstruction in rodent behavior studies.

Proposed method

• A multi-camera setup with one bottom-view and four side-view cameras was used to record 60 minutes of unconstrained mouse behavior at 50 Hz.
• 2D keypoint detection was performed using a ResNet-based DeepLabCut model trained on 1,000 labeled images with data augmentation to improve generalization.
• Geometric calibration was performed using a checkerboard target to establish projection matrices, enabling triangulation of 3D coordinates via polynomial fitting of depth from side-view projections.
• The triangulated 3D data was used to train two deep lift pose models: a linear residual network and a dilated temporal convolutional residual network.
• For model evaluation, data was shuffled across 10 random train/test splits (80%/20%), with early stopping and learning rate decay applied over 150 epochs.
• The temporal convolutional model was evaluated across varying temporal window sizes (15 to 243 timesteps) to determine optimal context length for 3D pose prediction.

Experimental results

Research questions

• RQ1 Can a geometrically calibrated multi-camera system produce reliable 3D pose data for freely moving rodents despite occlusions and limited camera visibility?
• RQ2 Which deep lift model—linear residual network or temporal convolutional network—performs better for 3D pose lifting from single-view 2D detections in rodent behavior?
• RQ3 What is the optimal temporal window size for the temporal convolutional model in terms of minimizing 3D pose estimation error?
• RQ4 How does the viewing perspective (e.g., bottom, side, rotated) affect the performance of deep lift models in 3D pose estimation?
• RQ5 Can the models generalize across different body part projections, especially in symmetric or ambiguous views?

Key findings

• The temporal convolutional model achieved a test loss of 0.45±0.003, significantly outperforming the linear residual model’s test loss of 1.47±0.031 on the (x,y)→z prediction task.
• The optimal temporal window size for the temporal convolutional model was 135 timesteps (2.7 seconds), yielding the lowest test loss across all settings.
• The (x,z)→y prediction direction at a 45° rotation around the x-axis yielded the lowest test loss for both models, indicating improved performance with diagonal viewing angles.
• The (x,y)→z prediction from the bottom view was the most accurately estimated by both models, suggesting that bottom-view 2D data provides strong priors for depth estimation.
• The temporal convolutional model showed superior generalization and robustness across all perspectives, with lower variance in test loss compared to the linear model.
• The optimal temporal window (2.7s) exceeded the 500ms decay time of pose auto-correlation, indicating that longer-range temporal dependencies enhance 3D pose prediction accuracy.