[Paper Review] PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Data

Roei Herzig, Ofir Abramovich+6 more|arXiv (Cornell University)|Dec 8, 2022

Human Pose and Action Recognition4 citations

TL;DR

PromptonomyViT proposes a multi-task prompt learning framework for video transformers that leverages synthetic scene data to improve real-world video understanding. By introducing task-specific prompts for auxiliary scene-level tasks—such as depth, segmentation, and normal estimation—the model captures shared spatio-temporal structures across synthetic tasks and enhances downstream action recognition without domain adaptation, achieving state-of-the-art performance on multiple benchmarks.

ABSTRACT

Action recognition models have achieved impressive results by incorporating scene-level annotations, such as objects, their relations, 3D structure, and more. However, obtaining annotations of scene structure for videos requires a significant amount of effort to gather and annotate, making these methods expensive to train. In contrast, synthetic datasets generated by graphics engines provide powerful alternatives for generating scene-level annotations across multiple tasks. In this work, we propose an approach to leverage synthetic scene data for improving video understanding. We present a multi-task prompt learning approach for video transformers, where a shared video transformer backbone is enhanced by a small set of specialized parameters for each task. Specifically, we add a set of "task prompts", each corresponding to a different task, and let each prompt predict task-related annotations. This design allows the model to capture information shared among synthetic scene tasks as well as information shared between synthetic scene tasks and a real video downstream task throughout the entire network. We refer to this approach as "Promptonomy", since the prompts model task-related structure. We propose the PromptonomyViT model (PViT), a video transformer that incorporates various types of scene-level information from synthetic data using the "Promptonomy" approach. PViT shows strong performance improvements on multiple video understanding tasks and datasets. Project page: \url{https://ofir1080.github.io/PromptonomyViT}

Motivation & Objective

• To address the high cost of annotating real video datasets with complex scene-level labels such as depth, segmentation, and 3D structure.
• To explore whether synthetic data with rich scene annotations can effectively improve video understanding models on real-world tasks.
• To develop a unified framework that enables video transformers to jointly learn from multiple synthetic scene tasks and a real downstream action recognition task.
• To minimize domain gap issues by integrating task-specific prompts that propagate information across the entire network.
• To demonstrate that prompt-based multi-task learning can outperform standard fine-tuning and domain adaptation techniques in low-resource settings.

Proposed method

• A shared Vision Transformer (ViT) backbone is used to process video clips, with task-specific learnable prompts injected at the input layer for each auxiliary scene-level task.
• Each task prompt is a small set of learnable parameters that attend to the video tokens and predict task-specific outputs such as depth maps, semantic segmentation, or normal vectors.
• The prompts are trained end-to-end using synthetic data from diverse sources (e.g., PHAV, HyperSim, KIST SynADL, EHOI), each providing different types of scene-level supervision.
• The action recognition head is trained on real data using the CLS token, while the task prompts are supervised by their respective synthetic labels, enabling knowledge transfer.
• The architecture enables cross-task information flow from early layers, allowing shared representations to emerge across synthetic tasks and benefit the real downstream task.
• The method is named 'Promptonomy' to reflect its design for managing inter-task structure and enabling multi-task learning in a unified transformer framework.

Experimental results

Research questions

• RQ1 Can synthetic scene data with multiple types of annotations improve video understanding models on real-world action recognition tasks?
• RQ2 How does multi-task prompt learning compare to standard fine-tuning or domain adaptation in reducing the domain gap between synthetic and real data?
• RQ3 To what extent can task-specific prompts capture shared representations across different synthetic scene-level tasks and transfer them to a real downstream task?
• RQ4 Does the inclusion of multiple auxiliary tasks (e.g., depth, segmentation, normal) lead to better generalization than single-task pretraining?
• RQ5 Can a unified prompt-based architecture effectively manage diverse scene-level tasks without requiring task-specific heads or domain adaptation modules?

Key findings

• PromptonomyViT achieves state-of-the-art performance on the Something-Something V2, Diving48, Ego4D, and AVA benchmarks, outperforming standard fine-tuning and domain adaptation baselines.
• The model shows significant accuracy gains on action recognition when pre-trained on synthetic data using multi-task prompts, even without explicit domain adaptation.
• Qualitative results demonstrate that task prompts generate meaningful and coherent predictions (e.g., depth, segmentation, hand-object boxes) on real video frames, despite never seeing real labels during training.
• The ablation study confirms that using multiple synthetic tasks with dedicated prompts leads to better performance than using only one or two tasks.
• The model generalizes well across diverse video domains, including egocentric, multi-view, and human-object interaction videos, due to the shared inductive bias from multi-task prompts.
• The use of task prompts enables effective knowledge transfer from synthetic to real data, with performance gains observed even in low-data regimes.