[Paper Review] Motion Style Transfer: Modular Low-Rank Adaptation for Deep Motion Forecasting

Parth Kothari, Danya Li+2 more|arXiv (Cornell University)|Nov 6, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes MoSA (Motion Style Transfer via Modular Low-Rank Adaptation), a parameter-efficient transfer learning method that adapts pre-trained motion forecasting models to new agent types or scenes with minimal data. By inserting low-rank adapters into frozen encoder layers and using a modular strategy to disentangle scene and motion features, MoSA achieves up to 25% lower generalization error with only 10–30 target samples, outperforming standard fine-tuning by 20% on the Level 5 dataset.

ABSTRACT

Deep motion forecasting models have achieved great success when trained on a massive amount of data. Yet, they often perform poorly when training data is limited. To address this challenge, we propose a transfer learning approach for efficiently adapting pre-trained forecasting models to new domains, such as unseen agent types and scene contexts. Unlike the conventional fine-tuning approach that updates the whole encoder, our main idea is to reduce the amount of tunable parameters that can precisely account for the target domain-specific motion style. To this end, we introduce two components that exploit our prior knowledge of motion style shifts: (i) a low-rank motion style adapter that projects and adjusts the style features at a low-dimensional bottleneck; and (ii) a modular adapter strategy that disentangles the features of scene context and motion history to facilitate a fine-grained choice of adaptation layers. Through extensive experimentation, we show that our proposed adapter design, coined MoSA, outperforms prior methods on several forecasting benchmarks.

Motivation & Objective

• To address poor generalization of deep motion forecasting models in low-data or out-of-distribution settings.
• To reduce the number of tunable parameters during adaptation while preserving performance on unseen agent types and scene contexts.
• To model motion style shifts—differences in navigation behavior—separately from invariant motion dynamics.
• To enable efficient transfer learning by focusing adaptation only on style-specific features rather than full network fine-tuning.
• To improve sample efficiency in transfer learning through modular, disentangled adaptation of scene context and motion history features.

Proposed method

• Introduces MoSA, a low-rank adapter module with two trainable matrices (A and B) forming a bottleneck to inject style-specific updates into frozen encoder layers.
• Applies MoSA as a parallel module to the attention mechanism, injecting style adjustments via low-rank updates to query and value matrices in self-attention layers.
• Employs a modular adaptation strategy that disentangles scene context and motion history encoders, allowing selective adaptation to only one stream.
• Uses a parameter initialization where matrix A is randomly initialized and B is zero-initialized, ensuring the original model behavior is preserved at start.
• Applies early stopping and one-cycle learning rate scheduling during adaptation to improve convergence and generalization.
• Employs a low-rank rank of 3 for MoSA in agent transfer and 8 in scene transfer, with learning rates tuned per method (e.g., 3e-3 for MoSA).

Experimental results

Research questions

• RQ1 Can a parameter-efficient adapter be designed to transfer motion forecasting models across different agent types with minimal data?
• RQ2 How does low-rank adaptation compare to full fine-tuning in terms of performance and parameter efficiency under distribution shifts?
• RQ3 Can disentangling scene context and motion history features improve the effectiveness of transfer learning in motion forecasting?
• RQ4 Does MoSA generalize across diverse scenarios such as pedestrian-to-cyclist, scene-to-scene, and city-region transfers?
• RQ5 What is the impact of adapter location (e.g., attention vs. feed-forward layers) on adaptation performance?

Key findings

• MoSA reduces the number of trainable parameters to less than 2% per layer while achieving state-of-the-art performance on SDD and inD datasets.
• On SDD and inD, MoSA improves generalization error by 25% when adapting with only 10–30 target trajectories.
• MoSA outperforms standard fine-tuning by 20% on the Level 5 dataset, even when adapting with just one pass over the unseen scene.
• Applying MoSA to attention layers (query and value matrices) yields better performance than applying it to feed-forward layers.
• The modular adaptation strategy enables significant performance gains in low-data regimes by selectively adapting only the motion history encoder when transferring across agent types.
• The initialization scheme (A random, B zero) ensures that MoSA does not alter the original model’s behavior at the start of training, enabling stable adaptation.