[Paper Review] Generating Long-term Trajectories Using Deep Hierarchical Networks

Stephan Zheng, Yisong Yue+1 more|arXiv (Cornell University)|Jun 21, 2017

Sports Analytics and PerformanceEconomics, Econometrics and Finance72 citations

TL;DR

The paper introduces a hierarchical policy network that jointly models macro-goals and micro-actions with an attention mechanism, enabling realistic long-term trajectory generation in basketball and outperforming non-hierarchical baselines. It demonstrates that incorporating macro-goals improves long-term planning and short-term action prediction.

ABSTRACT

We study the problem of modeling spatiotemporal trajectories over long time horizons using expert demonstrations. For instance, in sports, agents often choose action sequences with long-term goals in mind, such as achieving a certain strategic position. Conventional policy learning approaches, such as those based on Markov decision processes, generally fail at learning cohesive long-term behavior in such high-dimensional state spaces, and are only effective when myopic modeling lead to the desired behavior. The key difficulty is that conventional approaches are "shallow" models that only learn a single state-action policy. We instead propose a hierarchical policy class that automatically reasons about both long-term and short-term goals, which we instantiate as a hierarchical neural network. We showcase our approach in a case study on learning to imitate demonstrated basketball trajectories, and show that it generates significantly more realistic trajectories compared to non-hierarchical baselines as judged by professional sports analysts.

Motivation & Objective

• Motivate the need for long-term trajectory modeling in high-dimensional spatiotemporal settings.
• Propose a hierarchical policy class that separates macro-goals from micro-actions.
• Instantiate the policy as a hierarchical neural network with an attention mechanism guiding micro-actions via macro-goals.
• Evaluate the approach on professional basketball trajectory imitation and compare to non-hierarchical baselines and ground truth.

Proposed method

• Define a two-timescale policy framework with macro-goals g and micro-actions a over time.
• Model a raw micro-action u and synthesize the micro-action a from u and a macro-goal g using nonlinear transfer and synthesis functions (a = ψ(u, φ(g))).
• Implement an attention mechanism via a Hadamard product to integrate macro-goal guidance into the micro-action distribution (P(a|u,g) via P(raw)(u) and P(macro)(g)).
• Discretize state and goal spaces to enable tractable learning, using a 1-hot court occupancy representation and coarse macro-goals.
• Instantiate macro and micro policies as convolutional recurrent networks with an attention module that guides micro-actions toward macro-goals.
• Train the hierarchical network in multiple stages (micro, macro, attention) using weak labels for macro-goals and then fine-tune end-to-end.

Experimental results

Research questions

• RQ1 Can a hierarchical policy that separates macro-goals from micro-actions generate more realistic long-term trajectories than flat policies?
• RQ2 Does an attention-based interaction between macro-goals and micro-actions improve both long-term planning and short-term action accuracy?
• RQ3 How does the model perform on real expert trajectory data compared to non-hierarchical baselines and ground truth?
• RQ4 What is the impact of weak macro-goal labels on training and rollout quality?

Key findings

• The hierarchical policy network (HPN) generates rollouts with more realistic curved trajectories and longer-range planning toward macro-goals than baselines.
• HPN-attention improves micro-action prediction accuracy over non-hierarchical baselines in 4-step look-ahead tests.
• Human experts prefer HPN rollouts to baselines in most comparisons and view HPN as competitive with ground-truth trajectories.
• Weak macro-goal labels used during pre-training can enhance long-term rollout quality at the cost of some short-term prediction accuracy.
• The model provides interpretable dynamics showing macro-goals guiding the micro-planner through attention mechanisms.