[Paper Review] Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos

Ajay Kumar Tanwani, Pierre Sermanet+4 more|arXiv (Cornell University)|May 31, 2020

Human Pose and Action Recognition53 references4 citations

TL;DR

Motion2Vec proposes a semi-supervised representation learning method that clusters surgical video segments into a deep embedding space using metric learning in a Siamese network, with iterative refinement via a recurrent neural network for pseudo-labeling. It achieves 85.5% action segmentation accuracy and 0.94 cm position error in kinematic pose imitation on the da Vinci robot, outperforming state-of-the-art methods.

ABSTRACT

Learning meaningful visual representations in an embedding space can facilitate generalization in downstream tasks such as action segmentation and imitation. In this paper, we learn a motion-centric representation of surgical video demonstrations by grouping them into action segments/sub-goals/options in a semi-supervised manner. We present Motion2Vec, an algorithm that learns a deep embedding feature space from video observations by minimizing a metric learning loss in a Siamese network: images from the same action segment are pulled together while pushed away from randomly sampled images of other segments, while respecting the temporal ordering of the images. The embeddings are iteratively segmented with a recurrent neural network for a given parametrization of the embedding space after pre-training the Siamese network. We only use a small set of labeled video segments to semantically align the embedding space and assign pseudo-labels to the remaining unlabeled data by inference on the learned model parameters. We demonstrate the use of this representation to imitate surgical suturing motions from publicly available videos of the JIGSAWS dataset. Results give 85.5 % segmentation accuracy on average suggesting performance improvement over several state-of-the-art baselines, while kinematic pose imitation gives 0.94 centimeter error in position per observation on the test set. Videos, code and data are available at https://sites.google.com/view/motion2vec

Motivation & Objective

• To learn disentangled, motion-centric representations from surgical video demonstrations with minimal human annotation.
• To improve action segmentation and imitation learning performance using weakly supervised, self-supervised embedding learning.
• To enable generalization to new surgical tasks and real-world robot deployment with robust, interpretable representations.

Proposed method

• A Siamese network is trained with a contrastive loss to pull together images from the same action segment while pushing apart images from different segments.
• The model uses temporal ordering constraints to preserve spatio-temporal structure during embedding learning.
• After pre-training, a recurrent neural network infers pseudo-labels on unlabeled sequences to refine the embedding space.
• The embedding space is iteratively refined by retraining the Siamese network with pseudo-labeled data.
• A small set of labeled video segments is used to semantically align the embedding space and initialize the pseudo-labeling process.
• Pose imitation is performed using a feedforward network on embedded observations, enabling accurate kinematic replication on the da Vinci robot.

Experimental results

Research questions

• RQ1 Can a semi-supervised metric learning approach effectively group similar surgical action segments in a deep embedding space with minimal labeled data?
• RQ2 How does iterative pseudo-labeling via RNN improve the alignment and interpretability of learned representations in video-based imitation learning?
• RQ3 To what extent do Motion2Vec embeddings generalize to downstream tasks like action segmentation and kinematic pose imitation on real robotic systems?
• RQ4 How does the method compare to state-of-the-art metric and sequence learning baselines in terms of segmentation accuracy and robustness?
• RQ5 Can the learned representations preserve spatio-temporal structure under noise and domain shift?

Key findings

• Motion2Vec achieves 85.5% average segmentation accuracy on the JIGSAWS dataset, outperforming prior methods such as TCC (83.3%) and TCN (81.4%).
• The method shows robustness to input noise, maintaining accurate spatio-temporal alignment even with Gaussian noise (variance 0.15) on preprocessed images.
• Pose imitation using Motion2Vec embeddings yields a median position error of 0.94 cm per observation on the da Vinci robot, outperforming models trained on raw video.
• The per-surgeon pose imitation model trained on Motion2Vec embeddings achieves a median cosine quaternion loss of 36.49 and RMSE of 0.94 cm, indicating high fidelity in kinematic replication.
• Semi-supervised training with 25% or more labeled data consistently improves segmentation accuracy over fully supervised and unsupervised baselines.
• The iterative refinement loop using RNN-predicted pseudo-labels significantly enhances embedding quality and downstream performance.