[Paper Review] AOL: Adaptive Online Learning for Human Trajectory Prediction in Dynamic Video Scenes

Manh Huynh, Gita Alaghband|arXiv (Cornell University)|Feb 16, 2020

Video Surveillance and Tracking Methods21 references9 citations

TL;DR

This paper proposes AOL, an adaptive online learning framework that dynamically updates neural network weights for human trajectory prediction in dynamic video scenes. By integrating with LSTM and FPL models and using a least-recently-used strategy to maintain optimal weights, AOL boosts prediction accuracy by up to 50% (28% average) and sustains real-time inference at 20fps.

ABSTRACT

We present a novel adaptive online learning (AOL) framework to predict human movement trajectories in dynamic video scenes. Our framework learns and adapts to changes in the scene environment and generates best network weights for different scenarios. The framework can be applied to prediction models and improve their performance as it dynamically adjusts when it encounters changes in the scene and can apply the best training weights for predicting the next locations. We demonstrate this by integrating our framework with two existing prediction models: LSTM [3] and Future Person Location (FPL) [1]. Furthermore, we analyze the number of network weights for optimal performance and show that we can achieve real-time with a fixed number of networks using the least recently used (LRU) strategy for maintaining the most recently trained network weights. With extensive experiments, we show that our framework increases prediction accuracies of LSTM and FPL by ~17% and 28% on average, and up to ~50% for FPL on the worst case while achieving real-time (20fps).

Motivation & Objective

• Address the challenge of maintaining high prediction accuracy in dynamic video scenes where environmental conditions change over time.
• Overcome the limitations of static models that fail to adapt to scene shifts or new patterns in human motion.
• Develop a framework that continuously learns and selects the best-performing network weights for varying scene conditions.
• Enable real-time inference by efficiently managing a fixed set of pre-trained model weights using an LRU policy.
• Improve the performance of existing trajectory prediction models without requiring architectural changes.

Proposed method

• Design an adaptive online learning (AOL) framework that re-trains and updates model weights in real time as new scene data arrives.
• Integrate AOL with existing models—LSTM and Future Person Location (FPL)—to enhance their generalization across diverse dynamic environments.
• Apply a least-recently-used (LRU) strategy to maintain a fixed number of the most recently trained model weights for low-latency inference.
• Dynamically select the best-performing network weights from the cache based on recent performance in the current scene context.
• Use online learning to update model parameters incrementally, allowing rapid adaptation to scene changes without full retraining.
• Optimize the number of maintained model weights to balance accuracy and computational efficiency, achieving real-time performance at 20fps.

Experimental results

Research questions

• RQ1 Can an adaptive online learning framework improve trajectory prediction accuracy in dynamic video scenes with changing environmental conditions?
• RQ2 How does the performance of existing models like LSTM and FPL change when enhanced with adaptive weight selection and online learning?
• RQ3 What is the optimal number of model weights to maintain for real-time performance while maximizing prediction accuracy?
• RQ4 Can the LRU-based weight management strategy effectively balance accuracy and inference speed in dynamic scenes?
• RQ5 How does the framework perform under worst-case scene variations compared to baseline models?

Key findings

• AOL improves the average prediction accuracy of the FPL model by 28%, with a peak improvement of up to 50% in the worst-case scenario.
• The LSTM model sees an average accuracy gain of 17% when enhanced with the AOL framework.
• The framework sustains real-time inference at 20 frames per second using a fixed number of model weights managed via the LRU strategy.
• The number of maintained network weights is optimized to balance performance and computational cost, enabling efficient deployment.
• The integration of AOL with existing models is plug-and-play, requiring no architectural modifications.
• The LRU-based weight caching strategy effectively selects the most relevant and recent model weights, ensuring fast and accurate inference.