[Paper Review] Class-attention Video Transformer for Engagement Intensity Prediction

Xusheng Ai, Victor S. Sheng+2 more|arXiv (Cornell University)|Aug 12, 2022

Online Learning and Analytics4 citations

TL;DR

This paper proposes Class-attention Video Transformer (CavT), an end-to-end video transformer model that uses class-attention mechanisms to predict engagement intensity from video frames, achieving state-of-the-art performance. It further introduces Binary Order Representatives Sampling (BorS) to enhance training data diversity and frame utilization, reducing error to 0.0495 MSE on EmotiW-EP and 0.0377 MSE on DAiSEE.

ABSTRACT

In order to deal with variant-length long videos, prior works extract multi-modal features and fuse them to predict students' engagement intensity. In this paper, we present a new end-to-end method Class Attention in Video Transformer (CavT), which involves a single vector to process class embedding and to uniformly perform end-to-end learning on variant-length long videos and fixed-length short videos. Furthermore, to address the lack of sufficient samples, we propose a binary-order representatives sampling method (BorS) to add multiple video sequences of each video to augment the training set. BorS+CavT not only achieves the state-of-the-art MSE (0.0495) on the EmotiW-EP dataset, but also obtains the state-of-the-art MSE (0.0377) on the DAiSEE dataset. The code and models have been made publicly available at https://github.com/mountainai/cavt.

Motivation & Objective

• Address the challenge of predicting engagement intensity in long and variable-length online videos using end-to-end models.
• Overcome low frame utilization and poor generalization in existing video-based engagement prediction methods.
• Mitigate data scarcity and class imbalance issues in engagement prediction datasets through effective data augmentation.
• Adapt transformer architectures—specifically self-attention and class-attention mechanisms—for fine-grained engagement intensity regression in video.
• Develop a sampling strategy that improves model robustness and performance across diverse video lengths and content.

Proposed method

• Proposes Class-attention Video Transformer (CavT), which uses class-attention between a learnable class token and visual patches to aggregate spatiotemporal features for regression.
• Employs a standard ViT-style architecture with learnable position embeddings and multi-head self-attention over flattened video frames.
• Introduces Binary Order Representatives Sampling (BorS), a data augmentation method that divides video sequences into sliding windows and selects representative frames using binary-order logic.
• BorS adaptively generates multiple video sequences per input video by selecting frames from the middle or sub-windows in a binary-ordered fashion to ensure even distribution.
• Combines CavT with BorS to train a single end-to-end model that generalizes well across variable-length videos and improves performance on long sequences.
• Uses mean squared error (MSE) and mean absolute error (MAE) as evaluation metrics, with ablation studies on face extraction, class weighting, and sampling hyperparameters.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture with class-attention effectively predict continuous engagement intensity from raw video frames without handcrafted features?
• RQ2 How does the proposed BorS sampling strategy compare to random sampling in improving model generalization and performance on engagement prediction tasks?
• RQ3 To what extent does BorS enhance model performance when training data is limited or imbalanced?
• RQ4 Does the integration of face extraction improve engagement intensity prediction in the CavT framework?
• RQ5 What is the optimal number of sampling times (r) in BorS for different benchmark datasets like EmotiW-EP and DAiSEE?

Key findings

• BorS+CavT achieves the state-of-the-art MSE of 0.0495 on the EmotiW-EP dataset, outperforming prior methods.
• On the DAiSEE dataset, BorS+CavT achieves a new SOTA MSE of 0.0377, demonstrating strong generalization across diverse video lengths.
• The ablation study confirms that face extraction improves CavT performance, reducing MSE from 0.0759 to 0.0667 on EmotiW-EP.
• Weighted loss did not improve CavT performance and even increased MMSE, indicating that class imbalance is not effectively mitigated by loss weighting alone.
• Optimal sampling times for BorS are r=4 on EmotiW-EP and r=3 on DAiSEE, with performance degrading beyond these values due to overfitting.
• BorS outperforms random sampling by 0.0157 MSE (0.0495 vs. 0.0652), demonstrating the effectiveness of structured, binary-order frame selection.