[Paper Review] Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023

Daoji Huang, Otmar Hilliges+2 more|arXiv (Cornell University)|Jun 28, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

Palm proposes a vision-language and large language model-based framework for long-term action anticipation in egocentric videos, leveraging image captioning and action recognition to generate natural language prompts for LLMs. It achieves state-of-the-art performance on the Ego4D LTA challenge, ranking first with an action edit distance of 0.8856 on the test set.

ABSTRACT

We present Palm, a solution to the Long-Term Action Anticipation (LTA) task utilizing vision-language and large language models. Given an input video with annotated action periods, the LTA task aims to predict possible future actions. We hypothesize that an optimal solution should capture the interdependency between past and future actions, and be able to infer future actions based on the structure and dependency encoded in the past actions. Large language models have demonstrated remarkable commonsense-based reasoning ability. Inspired by that, Palm chains an image captioning model and a large language model. It predicts future actions based on frame descriptions and action labels extracted from the input videos. Our method outperforms other participants in the EGO4D LTA challenge and achieves the best performance in terms of action prediction. Our code is available at https://github.com/DanDoge/Palm

Motivation & Objective

• To address the challenge of long-term action anticipation in egocentric videos, where future actions are uncertain and multiple plausible sequences exist.
• To overcome limitations of previous models that rely solely on shared features and independent verb/noun classification.
• To leverage commonsense reasoning from large language models by encoding past actions and visual context in natural language.
• To improve prediction by modeling dependencies between past and future actions through structured language prompts.
• To achieve robust and diverse action predictions using few-shot prompting with MMR-based exemplar selection.

Proposed method

• The method uses an image captioning model to generate descriptive narrations from the middle frame of each past action segment, starting with 'A person is'.
• An action recognition model based on EgoVLP and a Transformer extracts (verb, noun) pairs from video clips.
• Past action descriptions and labels are combined into a natural language prompt, formatted as a sentence completion task for the LLM.
• A few-shot prompt template includes instruction, exemplars with past actions and future sequences, and a query with past actions only.
• Exemplars are selected using MMR-based strategy to balance semantic relevance to the query and diversity, using MPNet embeddings and cosine similarity.
• Multiple future action sequences are generated by sampling from the LLM output, and the best sequence is selected based on minimum edit distance to ground truth.

Experimental results

Research questions

• RQ1 Can large language models improve long-term action anticipation by leveraging commonsense knowledge and structural dependencies in action sequences?
• RQ2 How effective is using natural language prompts derived from image captions and action labels for LLM-based action prediction?
• RQ3 What is the impact of prompt design, including exemplar selection and model size, on prediction accuracy and diversity?
• RQ4 How do visual context (captions) and action labels individually contribute to prediction performance?
• RQ5 To what extent is model performance limited by the accuracy of the past action recognition module?

Key findings

• Palm achieved the best performance in the Ego4D Long-Term Action Anticipation Challenge 2023, with an action edit distance of 0.8856 on the test set.
• Using BLIP2 for captioning outperformed vit-gpt2, especially in noun prediction, reducing noun edit distance to 0.6767.
• Incorporating MMR-based exemplar selection improved performance over random selection, reducing action edit distance to 0.8934.
• Including both narrations and action labels in the prompt yielded the best results, with action edit distance of 0.8934, outperforming either modality alone.
• Larger language models like GPT-Neo-1.3B with larger context windows achieved better results than smaller GPT-2 variants, with lower edit distances.
• There is a strong negative correlation between past action recognition accuracy and edit distance, indicating that model performance is highly sensitive to recognition quality.