[Paper Review] Vehicle Trajectory Prediction by Transfer Learning of Semi-Supervised Models

Nick Lamm, Shashank Jaiprakash+2 more|arXiv (Cornell University)|Jul 13, 2020

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper demonstrates that transfer learning using semi-supervised models significantly improves vehicle trajectory prediction performance over supervised models on real-world benchmarks. By leveraging large-scale unlabeled data—scaling pre-training from millions to a billion images—it shows that contrastive learning (SimCLR) outperforms teacher-student methods and boosts performance across both low-level (camera input) and mid-level (map-annotated) representations, with gains up to 40.1% on multi-modal metrics.

ABSTRACT

In this work we show that semi-supervised models for vehicle trajectory prediction significantly improve performance over supervised models on state-of-the-art real-world benchmarks. Moving from supervised to semi-supervised models allows scaling-up by using unlabeled data, increasing the number of images in pre-training from Millions to a Billion. We perform ablation studies comparing transfer learning of semi-supervised and supervised models while keeping all other factors equal. Within semi-supervised models we compare contrastive learning with teacher-student methods as well as networks predicting a small number of trajectories with networks predicting probabilities over a large trajectory set. Our results using both low-level and mid-level representations of the driving environment demonstrate the applicability of semi-supervised methods for real-world vehicle trajectory prediction.

Motivation & Objective

• To evaluate whether semi-supervised pre-training improves trajectory prediction performance compared to supervised pre-training under identical conditions.
• To investigate the effectiveness of different semi-supervised learning techniques—contrastive learning versus teacher-student distillation—for driving scene representation learning.
• To compare performance gains when using semi-supervised models for predicting a small set of trajectories versus a large fixed set of candidate trajectories.
• To validate the generalizability of semi-supervised representations across both low-level (camera images) and mid-level (annotated map images) driving environment representations.

Proposed method

• Fine-tune pre-trained semi-supervised and supervised models (e.g., SimCLR, ResNeXt-101 SSL, SWSL) on vehicle trajectory prediction tasks using fixed architectures and computational resources.
• Use contrastive learning (SimCLR) to pre-train models on a billion-image dataset of unlabeled images, enabling self-supervised feature learning.
• Apply knowledge distillation via teacher-student frameworks (e.g., ResNeXt-101 32x4d SSL, SWSL) to learn from noisy student signals on large-scale unlabeled data.
• Evaluate models on two benchmark datasets: Drive360 (low-level, camera-based) and nuScenes (mid-level, map-annotated), using both uni-modal and multi-modal trajectory prediction metrics.
• Downsample training data and input resolution (e.g., 1920x1080 → 160x90) to reduce training time while preserving performance evaluation on full test splits.
• Freeze 3/4 of the lowest network blocks during fine-tuning to preserve pre-trained features while adapting the final layers to the downstream prediction task.

Experimental results

Research questions

• RQ1 Does transfer learning with semi-supervised models improve trajectory prediction performance compared to supervised models when all other factors (architecture, data, compute) are held constant?
• RQ2 How do different semi-supervised learning techniques—contrastive learning (SimCLR) versus teacher-student distillation (SWSL, SSL)—compare in performance on vehicle trajectory prediction tasks?
• RQ3 Does the benefit of semi-supervised pre-training vary depending on whether the model predicts a small number of trajectories or a large fixed set of candidate trajectories?
• RQ4 Can semi-supervised representations generalize effectively to both low-level (camera input) and mid-level (map-annotated) representations of the driving environment?

Key findings

• Semi-supervised models outperform supervised models on both low-level and mid-level representations, with SimCLR achieving the best overall performance across all benchmarks.
• Contrastive learning (SimCLR) significantly outperforms teacher-student methods (ResNeXt-101 32x4d SSL and SWSL), with the former reducing steering angle MSE by 0.99 and speed MSE by 0.87 compared to supervised ResNet-50 on Drive360.
• On multi-modal trajectory prediction with CoverNet, semi-supervised models improved performance by up to 40.1% over supervised baselines, demonstrating strong gains in handling trajectory uncertainty.
• For small-trajectory-set prediction with MTP, semi-supervised models improved uni-modal metrics by up to 17.3%, but showed no significant gains on multi-modal metrics, indicating task-dependent benefits.
• The ResNeXt-101 32x4d SWSL model underperformed the supervised ResNet-101 baseline on Drive360, suggesting that not all semi-supervised methods generalize equally well across tasks.
• The performance gains from semi-supervised pre-training are achieved without additional inference or training compute, making it a plug-and-play improvement for existing trajectory prediction pipelines.