[Paper Review] Improving ProtoNet for Few-Shot Video Object Recognition: Winner of ORBIT Challenge 2022

Li Gu, Zhixiang Chi+3 more|arXiv (Cornell University)|Oct 1, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper presents the winning solution for the ORBIT 2022 Few-Shot Video Object Recognition Challenge, improving ProtoNet through three key techniques: embedding adaptation using a transformer encoder to enhance prototype discriminability, uniform video clip sampling to ensure consistent temporal coverage and reduce sampling bias, and invalid frame detection via edge detection to filter out low-information frames. The method achieves a 5.39% absolute accuracy gain over the baseline, reaching 71.69% frame-level accuracy on the ORBIT benchmark.

ABSTRACT

In this work, we present the winning solution for ORBIT Few-Shot Video Object Recognition Challenge 2022. Built upon the ProtoNet baseline, the performance of our method is improved with three effective techniques. These techniques include the embedding adaptation, the uniform video clip sampler and the invalid frame detection. In addition, we re-factor and re-implement the official codebase to encourage modularity, compatibility and improved performance. Our implementation accelerates the data loading in both training and testing.

Motivation & Objective

• To address the challenge of few-shot video object recognition in real-world, noisy settings, particularly for vision-impaired users who provide low-quality, variable-length video clips.
• To improve the robustness and accuracy of ProtoNet-based models when faced with noisy, cluttered video inputs and variable-length sequences.
• To enhance prototype quality and sampling efficiency in meta-learning frameworks for video recognition by introducing adaptive, uniform, and filtering mechanisms.
• To optimize the data pipeline for improved training and inference speed, supporting reproducibility and extensibility in future research.

Proposed method

• Embedding adaptation is applied to the support set prototypes using a single-layer transformer encoder that models inter-prototype relationships, enhancing discriminative representation for each episode.
• A uniform video clip sampler replaces random sampling to ensure consistent temporal coverage across videos of varying lengths, reducing over-sampling in short videos and under-sampling in long ones.
• Invalid frame detection is implemented using a pre-trained edge detector (e.g., Canny) with an empirical threshold to identify and discard frames with minimal content, such as background-only frames.
• The method integrates data augmentation to enrich the diversity of sampled clips and improve generalization.
• The original ORBIT codebase is refactored and optimized, achieving over 2.7x faster data loading during both training and inference.
• The full pipeline uses EfficientNet-B0 as the feature extractor and applies episodic training with cosine similarity for classification, following the LITE framework.

Experimental results

Research questions

• RQ1 How can prototype quality in few-shot video recognition be improved when support videos are short, noisy, and of variable length?
• RQ2 What sampling strategy minimizes distributional bias and maximizes temporal coverage in variable-length video sequences during few-shot learning?
• RQ3 To what extent can edge-based invalid frame detection reduce the impact of redundant or background-only frames on model performance?
• RQ4 Can the integration of embedding adaptation via a transformer encoder improve inter-class discriminability in episodic learning?
• RQ5 How much performance gain can be achieved by optimizing the data loading pipeline in a real-world few-shot video benchmark?

Key findings

• The proposed method achieves a 5.39% absolute improvement in frame-level accuracy over the baseline ProtoNet, reaching 71.69% on the ORBIT benchmark.
• Embedding adaptation alone contributes 2.87 percentage points to the accuracy gain by enhancing prototype discriminability through inter-prototype attention.
• Uniform video clip sampling improves accuracy by 1.52 percentage points by reducing sampling bias and ensuring consistent temporal coverage.
• Invalid frame detection via edge detection adds 0.12 percentage points to the accuracy by filtering out low-information frames.
• The refactored data pipeline accelerates data loading by over 2.7x during both training and testing, significantly improving training efficiency.
• The method achieves the highest accuracy among 12 submissions in the ORBIT 2022 challenge, outperforming all other entries on 11 out of 17 users.