[Paper Review] Multi-Task Learning of Object State Changes from Uncurated Videos

Tomáš Souček, Jean-Baptiste Alayrac+3 more|arXiv (Cornell University)|Nov 24, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes a multi-task self-supervised learning framework that jointly localizes object states and state-modifying actions in long, uncurated web videos using only noisy video-level supervision. By leveraging cross-task constraints and an end-to-end trainable architecture, the method achieves a 40% relative improvement over prior single-task approaches on the ChangeIt dataset and demonstrates strong zero-shot generalization to egocentric videos.

ABSTRACT

We aim to learn to temporally localize object state changes and the corresponding state-modifying actions by observing people interacting with objects in long uncurated web videos. We introduce three principal contributions. First, we explore alternative multi-task network architectures and identify a model that enables efficient joint learning of multiple object states and actions such as pouring water and pouring coffee. Second, we design a multi-task self-supervised learning procedure that exploits different types of constraints between objects and state-modifying actions enabling end-to-end training of a model for temporal localization of object states and actions in videos from only noisy video-level supervision. Third, we report results on the large-scale ChangeIt and COIN datasets containing tens of thousands of long (un)curated web videos depicting various interactions such as hole drilling, cream whisking, or paper plane folding. We show that our multi-task model achieves a relative improvement of 40% over the prior single-task methods and significantly outperforms both image-based and video-based zero-shot models for this problem. We also test our method on long egocentric videos of the EPIC-KITCHENS and the Ego4D datasets in a zero-shot setup demonstrating the robustness of our learned model.

Motivation & Objective

• To address the challenge of learning object state changes and corresponding actions from long, uncurated web videos without human-annotated temporal boundaries.
• To enable efficient joint learning across multiple related interaction tasks, such as cutting different fruits or pouring liquids, by sharing representations across tasks.
• To design a self-supervised learning procedure that exploits structural constraints between object states and actions to enable end-to-end training from weak supervision.
• To evaluate the model’s zero-shot transferability to egocentric video datasets like EPIC-KITCHENS and Ego4D, despite training on third-person web videos.

Proposed method

• The method employs a multi-task neural network architecture that jointly predicts object states (e.g., whole, sliced) and state-modifying actions (e.g., cutting, pouring) for each video clip.
• It introduces a pseudo-label sampling strategy that generates training signals by enforcing logical constraints, such as ensuring only one action is predicted per video and that actions do not occur when an object is already in its final state.
• The model is trained end-to-end using only video-level labels from search queries (e.g., 'How to cut a pineapple'), without any frame-level annotations.
• Cross-task consistency is enforced via self-supervision: for example, the model learns that pouring coffee and pouring water both result in a cup transitioning from empty to full.
• The backbone is a single image encoder (e.g., Swin-T) fine-tuned end-to-end, avoiding the need for frozen features or separate models per task.
• The framework is evaluated on large-scale datasets including ChangeIt and COIN, with additional zero-shot testing on EPIC-KITCHENS and Ego4D.

Experimental results

Research questions

• RQ1 What multi-task network architecture enables effective information sharing across diverse object state changes and actions?
• RQ2 How can self-supervised learning be designed to exploit structural constraints between object states and actions in the absence of temporal annotations?
• RQ3 Can a model trained on noisy, uncurated web videos generalize effectively to egocentric video datasets in a zero-shot setting?
• RQ4 How does joint multi-task learning compare to single-task baselines in terms of precision and robustness on long, uncurated videos?

Key findings

• The proposed multi-task model achieves a 40% relative improvement in performance over prior single-task methods on the ChangeIt dataset, with 14 percentage points higher state precision and 12 percentage points higher action precision.
• The model significantly outperforms both image-based and video-based zero-shot models on the same benchmark, demonstrating the advantage of joint learning with self-supervision.
• End-to-end fine-tuning of the backbone leads to a substantial performance boost compared to using frozen features, even with only a single image encoder.
• The model generalizes robustly to egocentric videos in a zero-shot setup, outperforming previous works on EPIC-KITCHENS and Ego4D despite domain shift from third-person to egocentric views.
• Qualitative results show the model correctly retrieves relevant videos, localizes initial, action, and end states, and maintains consistency across different video examples.
• The model effectively filters out irrelevant videos (with low classification scores) and maintains visual consistency in predicted states across diverse video clips.