[Paper Review] AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

Qi Zhao, Shijie Wang+6 more|arXiv (Cornell University)|Jul 31, 2023

Action Observation and Synchronization8 citations

TL;DR

AntGPT leverages large language models to infer goals and model temporal dynamics for long-term action anticipation from videos, achieving state-of-the-art results on multiple benchmarks and enabling a compact distilled model.

ABSTRACT

Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing what commonly happens after his/her current action (e.g. crack eggs)? What if we also know the longer-term goal of the actor (e.g. making egg fried rice)? The long-term action anticipation (LTA) task aims to predict an actor's future behavior from video observations in the form of verb and noun sequences, and it is crucial for human-machine interaction. We propose to formulate the LTA task from two perspectives: a bottom-up approach that predicts the next actions autoregressively by modeling temporal dynamics; and a top-down approach that infers the goal of the actor and plans the needed procedure to accomplish the goal. We hypothesize that large language models (LLMs), which have been pretrained on procedure text data (e.g. recipes, how-tos), have the potential to help LTA from both perspectives. It can help provide the prior knowledge on the possible next actions, and infer the goal given the observed part of a procedure, respectively. To leverage the LLMs, we propose a two-stage framework, AntGPT. It first recognizes the actions already performed in the observed videos and then asks an LLM to predict the future actions via conditioned generation, or to infer the goal and plan the whole procedure by chain-of-thought prompting. Empirical results on the Ego4D LTA v1 and v2 benchmarks, EPIC-Kitchens-55, as well as EGTEA GAZE+ demonstrate the effectiveness of our proposed approach. AntGPT achieves state-of-the-art performance on all above benchmarks, and can successfully infer the goal and thus perform goal-conditioned "counterfactual" prediction via qualitative analysis. Code and model will be released at https://brown-palm.github.io/AntGPT

Motivation & Objective

• Investigate whether large language models (LLMs) encode useful prior knowledge for long-term action anticipation (LTA) from video data.
• Explore bottom-up (temporal dynamics) and top-down (goal-conditioned) LTA frameworks leveraging LLMs.
• Assess whether goals can be inferred from observed actions using in-context learning and minimal supervision.
• Demonstrate that LLMs can be distilled into a compact neural network for efficient LTA inference.

Proposed method

• Represent videos as sequences of noisy action labels derived from an action recognizer to interface with LLMs.
• Use in-context learning (ICL) with LLMs to infer long-term goals from observed action sequences.
• Implement bottom-up LTA by autoregressively or parallely predicting future actions from observed action tokens via a Transformer-based temporal model.
• Implement top-down LTA by conditioning action planning on goals inferred by the LLM (goal-conditioned planning).
• Explore LLMs as temporal dynamics models by fine-tuning or prompting to perform sequence completion of actions, with or without explicit goal conditioning.
• Apply knowledge distillation to compress LLM-encoded temporal dynamics into a compact student network (1.3% of the original size).
• Evaluate few-shot ICL and chain-of-thought prompts (CoT) for goal inference and LTA.

Experimental results

Research questions

• RQ1 Does top-down, goal-conditioned LTA outperform bottom-up approaches?
• RQ2 Can LLMs infer useful goals for top-down LTA with minimal supervision via in-context learning?
• RQ3 Do LLMs capture prior knowledge for temporal dynamics of actions, and what is an effective interface between video and LLMs?
• RQ4 Can the LLM-based knowledge be distilled into a compact model for efficient inference?

Key findings

• LLM-informed goal conditioning improves top-down LTA across Ego4D, EK-55, and EGTEA benchmarks, especially for rare actions.
• LLMs can infer helpful high-level goals from observed actions via in-context learning, leading to improved LTA when goals are used.
• LLMs can model temporal dynamics of actions as sequence completion, with competitive results in few-shot settings.
• Explicit goal inference is not strictly necessary in some setups, as LLMs can implicitly infer goals while predicting future actions.
• A distilled 91M-parameter model (1.3% of 7B) can match or surpass the full-size teacher in temporal modeling performance, confirming knowledge distillation feasibility.