[Paper Review] Elaborative Rehearsal for Zero-shot Action Recognition

Shizhe Chen, Dong Huang|arXiv (Cornell University)|Aug 5, 2021

Human Pose and Action Recognition51 references4 citations

TL;DR

This paper proposes an Elaborative Rehearsal (ER)-enhanced zero-shot action recognition model that improves generalization by generating discriminative Elaborative Descriptions (EDs) for action classes and incorporating Elaborative Concepts (ECs) from detected objects in videos. By leveraging pre-trained BERT for ED encoding and a joint video feature learning framework with temporal and object streams, the model achieves state-of-the-art performance on multiple benchmarks and demonstrates strong generalization, outperforming 1-shot supervised baselines on Kinetics ZSAR.

ABSTRACT

The growing number of action classes has posed a new challenge for video understanding, making Zero-Shot Action Recognition (ZSAR) a thriving direction. The ZSAR task aims to recognize target (unseen) actions without training examples by leveraging semantic representations to bridge seen and unseen actions. However, due to the complexity and diversity of actions, it remains challenging to semantically represent action classes and transfer knowledge from seen data. In this work, we propose an ER-enhanced ZSAR model inspired by an effective human memory technique Elaborative Rehearsal (ER), which involves elaborating a new concept and relating it to known concepts. Specifically, we expand each action class as an Elaborative Description (ED) sentence, which is more discriminative than a class name and less costly than manual-defined attributes. Besides directly aligning class semantics with videos, we incorporate objects from the video as Elaborative Concepts (EC) to improve video semantics and generalization from seen actions to unseen actions. Our ER-enhanced ZSAR model achieves state-of-the-art results on three existing benchmarks. Moreover, we propose a new ZSAR evaluation protocol on the Kinetics dataset to overcome limitations of current benchmarks and demonstrate the first case where ZSAR performance is comparable to few-shot learning baselines on this more realistic setting. We will release our codes and collected EDs at https://github.com/DeLightCMU/ElaborativeRehearsal.

Motivation & Objective

• To address the challenge of zero-shot action recognition (ZSAR) where no training data is available for unseen actions.
• To improve semantic representation of action classes beyond ambiguous word embeddings or manual attributes.
• To enhance video semantic representations by integrating object-level concepts from pre-trained image models.
• To develop a training objective that mimics human memory rehearsal for better generalization to unseen actions.
• To establish a more realistic ZSAR benchmark on Kinetics to compare with few-shot learning baselines.

Proposed method

• Constructs Elaborative Descriptions (EDs) for each action class using pre-trained BERT to generate more discriminative, context-rich textual representations than simple class names.
• Detects objects in videos using a pre-trained image classifier and uses their class names as Elaborative Concepts (ECs) to enrich video semantics.
• Employs a dual-stream network to jointly encode spatio-temporal video features (via TSM) and object features (via ResNet), with a multimodal channel attention mechanism to fuse them.
• Introduces an Elaborative Rehearsal (ER) loss that encourages the model to align video features with both action EDs and ECs, enhancing generalization through semantic rehearsal.
• Uses a shared embedding space for EDs and ECs, enabling cross-modal alignment and improved transferability to unseen actions.
• Trains the model using a contrastive loss on video-action pairs and the ER loss to jointly optimize for zero-shot generalization.

Figure 1 : Attributes and word embeddings are insufficient to semantically represent action classes. Our Elaborative Rehearsal approach defines actions by Elaborative Descriptions (EDs) and associates videos with Elaborative Concepts (ECs, known concepts detected from the video), which improve video

Experimental results

Research questions

• RQ1 Can more expressive textual descriptions (EDs) improve zero-shot action recognition beyond simple word embeddings?
• RQ2 Can object-level semantic concepts (ECs) from image models enhance video representation and generalization in ZSAR?
• RQ3 Does a training objective inspired by human memory rehearsal (ER loss) improve generalization to unseen actions?
• RQ4 How does the proposed ZSAR model compare to few-shot supervised learning in realistic settings?
• RQ5 Can the proposed method generalize effectively across diverse action classes without any labeled training examples?

Key findings

• The ER-enhanced ZSAR model achieves state-of-the-art performance on Olympic Sports, HMDB51, and UCF101 benchmarks, with top-1 accuracy of 87.2% on UCF101.
• Using BERT to encode EDs significantly improves performance over GloVe-based pooling methods, demonstrating the benefit of contextual understanding in action descriptions.
• The ER loss boosts zero-shot generalization by a large margin, especially on unseen actions, showing its effectiveness in semantic rehearsal.
• Combining spatio-temporal (ST) and object features via multimodal attention yields the best performance, outperforming either modality alone.
• The model outperforms the 1-shot supervised baseline on Kinetics ZSAR, achieving 42.1% top-1 accuracy with zero training data, though it still lags behind 2-shot supervised models.
• EDs are beneficial even for object representations, as replacing object class names with their EDs improves performance, indicating universal utility of elaborative representations.

Figure 2 : Architecture of our ER-enhanced ZSAR model. The action class embedding function (left) extends action names to EDs towards action class embedding $z$ . The multimodal video embedding function generates spatio-temporal and object features $[x_{vo},x_{ov}]$ (middle) for the video. The ER lo