[Paper Review] Opening the Vocabulary of Egocentric Actions

Dibyadip Chatterjee, Fadime Şener+2 more|arXiv (Cornell University)|Aug 22, 2023

Human Pose and Action Recognition4 citations

TL;DR

This paper introduces open vocabulary action recognition for egocentric videos, where models generalize verbs to novel objects using an object-agnostic verb encoder and a CLIP-based prompt learning method for active objects. It achieves state-of-the-art performance on EPIC-KITCHENS-100 and Assembly101 benchmarks, significantly outperforming closed-set methods in recognizing unseen verb-object compositions.

ABSTRACT

Human actions in egocentric videos are often hand-object interactions composed from a verb (performed by the hand) applied to an object. Despite their extensive scaling up, egocentric datasets still face two limitations - sparsity of action compositions and a closed set of interacting objects. This paper proposes a novel open vocabulary action recognition task. Given a set of verbs and objects observed during training, the goal is to generalize the verbs to an open vocabulary of actions with seen and novel objects. To this end, we decouple the verb and object predictions via an object-agnostic verb encoder and a prompt-based object encoder. The prompting leverages CLIP representations to predict an open vocabulary of interacting objects. We create open vocabulary benchmarks on the EPIC-KITCHENS-100 and Assembly101 datasets; whereas closed-action methods fail to generalize, our proposed method is effective. In addition, our object encoder significantly outperforms existing open-vocabulary visual recognition methods in recognizing novel interacting objects.

Motivation & Objective

• To address the sparsity of verb-object compositions in egocentric action datasets, which limits generalization to novel objects.
• To enable action recognition models to generalize verbs to open-vocabulary objects not seen during training.
• To overcome biases from long-tailed distributions and static frame representations in egocentric video understanding.
• To develop a method that recognizes active objects undergoing state changes, even in cluttered scenes with distractors.
• To create realistic open-vocabulary benchmarks on EPIC-KITCHENS-100 and Assembly101 for evaluating generalization beyond base classes.

Proposed method

• Decouples verb and object prediction using an Object-Agnostic Retraining (OAP) strategy with a supervised contrastive loss and task-specific guiding augmentations for egocentric videos.
• Designs guiding augmentations that preserve temporal motion and object interactions, improving robustness to camera motion and distractors.
• Proposes Active Object Prompting (AOP), a learnable, verb-conditioned prompt mechanism that guides CLIP to identify active objects in egocentric scenes.
• Fine-tunes CLIP’s text encoder with learnable word embeddings per object class, improving alignment between visual features and open-vocabulary object labels.
• Uses a two-branch architecture: a verb encoder trained on OAP to learn object-agnostic representations, and an object encoder using CLIP with AOP for zero-shot generalization.
• Creates open-vocabulary benchmarks by repurposing existing splits in EPIC-KITCHENS-100 and Assembly101, introducing novel object classes at inference time.

Experimental results

Research questions

• RQ1 Can verb representations be disentangled from object-specific features to enable generalization to novel objects in egocentric videos?
• RQ2 How can active objects—those undergoing state changes—be reliably recognized in cluttered egocentric scenes with distractors?
• RQ3 To what extent can CLIP-based visual-linguistic models be adapted via prompt learning to recognize novel interacting objects in egocentric action recognition?
• RQ4 Does object-agnostic training with guiding augmentations improve zero-shot generalization of verbs to unseen objects?
• RQ5 How effective is the proposed method in open-vocabulary settings compared to closed-set baselines on real-world egocentric datasets?

Key findings

• The proposed method achieves a Harmonized Mean (HM) of 10.8 on the HOI crop ablation, significantly outperforming the full-frame baseline (HM=7.5), demonstrating the effectiveness of localized object and hand cropping.
• With m=5 learnable word embeddings per object, the model achieves the best novel class performance, indicating that fine-tuning the prompt vocabulary improves generalization to unseen objects.
• The object-agnostic verb encoder with guiding augmentations improves generalization, as shown by higher performance on novel object compositions compared to standard contrastive learning.
• Active Object Prompting (AOP) enables the CLIP model to correctly identify the active object in complex egocentric scenes, even when multiple objects are present and only one undergoes a state change.
• Interpretability analysis shows that fine-tuned prompts often correct for plurality (e.g., 'eggs' instead of 'egg') or synonymy (e.g., 'faucet' instead of 'tap'), improving alignment with natural language.
• The method significantly outperforms closed-action recognition baselines on both EPIC-KITCHENS-100 and Assembly101, proving its effectiveness in open-vocabulary generalization.