[Paper Review] A Graph Attention Based Approach for Trajectory Prediction in Multi-agent Sports Games

Ding Ding, H. Howie Huang|arXiv (Cornell University)|Dec 18, 2020

Sports Analytics and Performance19 references4 citations

TL;DR

This paper proposes a graph attention network with a Temporal Convolutional Network (TCN) for multi-agent trajectory prediction in sports games, enabling fine-grained modeling of player dependencies. It achieves state-of-the-art performance by learning team-specific strategies through permutation-equivariant graph attention and long-context temporal modeling, outperforming RNN-based and GNN-based baselines on basketball and soccer datasets.

ABSTRACT

This work investigates the problem of multi-agents trajectory prediction. Prior approaches lack of capability of capturing fine-grained dependencies among coordinated agents. In this paper, we propose a spatial-temporal trajectory prediction approach that is able to learn the strategy of a team with multiple coordinated agents. In particular, we use graph-based attention model to learn the dependency of the agents. In addition, instead of utilizing the recurrent networks (e.g., VRNN, LSTM), our method uses a Temporal Convolutional Network (TCN) as the sequential model to support long effective history and provide important features such as parallelism and stable gradients. We demonstrate the validation and effectiveness of our approach on two different sports game datasets: basketball and soccer datasets. The result shows that compared to related approaches, our model that infers the dependency of players yields substantially improved performance. Code is available at https://github.com/iHeartGraph/predict

Motivation & Objective

• To address the limitation of prior methods in capturing fine-grained, role-specific dependencies among coordinated agents in multi-agent sports games.
• To develop a trajectory prediction model that learns team-specific strategies from raw trajectory data without predefined roles.
• To replace recurrent networks (e.g., LSTM, VRNN) with a Temporal Convolutional Network (TCN) for better gradient stability and long-context modeling.
• To enable permutation-equivariant modeling by using a fully connected graph structure where agent order does not affect predictions.
• To validate the model’s ability to generalize across teams and specialize on single-team strategies through empirical evaluation.

Proposed method

• Uses a fully connected graph to represent agents at each time step, ensuring permutation equivariance by treating players symmetrically.
• Applies a graph attention mechanism to compute attention coefficients that model dynamic, fine-grained dependencies between agents based on their spatial positions and states.
• Employs a Temporal Convolutional Network (TCN) as the sequential model to capture long-range temporal dependencies with stable gradients and parallelism.
• Processes agent features through an attention block to generate a graph embedding at each time step, which is then fed into residual blocks of the TCN.
• Trains the entire model end-to-end to predict future trajectories of all agents, including the ball, using a multi-head attention mechanism for improved representation learning.
• Uses a residual block design in the TCN to allow deep network training by learning residual mappings.

Experimental results

Research questions

• RQ1 Can a graph attention mechanism effectively model fine-grained dependencies between players in multi-agent sports games?
• RQ2 Does replacing recurrent networks with a TCN improve trajectory prediction performance in terms of accuracy, training stability, and long-context modeling?
• RQ3 Can the model learn and infer team-specific strategies from trajectory data without prior role assignment?
• RQ4 How does the performance of the proposed method compare to state-of-the-art approaches on both basketball and soccer datasets?
• RQ5 Does the model achieve better performance when trained on a single team versus mixed teams?

Key findings

• The proposed Graph + Attention + TCN model achieves the lowest average L2 error of 4.95 ft on the Toronto Raptors defense dataset and 6.20 ft on the offense dataset, outperforming all baselines.
• On the defense prediction task, the model reduces the miss rate to 48.44%, significantly lower than the 63.01% of the next best method (Graph + VRNNs).
• For the offense prediction task, the model achieves a 59.44% miss rate, compared to 63.21% for Graph + VRNNs, demonstrating improved generalization and strategy learning.
• When modeling a single team (Toronto Raptors), the attention-based models show a substantial performance gain over non-attention baselines, indicating effective learning of team-specific strategies.
• The TCN-based sequential model enables stable training and long effective history, with better gradient flow and parallelization compared to RNNs like LSTM.
• The ablation study confirms that the graph attention mechanism is critical for performance, as removing it leads to a significant drop in accuracy, especially in complex coordination scenarios.