[Paper Review] Determining the best attributes for surveillance video keywords generation

Liangchen Liu, Arnold Wiliem+3 more|arXiv (Cornell University)|Feb 21, 2016

Video Analysis and Summarization3 citations

TL;DR

This paper proposes a novel method to automatically select the best attribute discovery approach for surveillance video keyword generation by leveraging shared structural patterns in meaningful attributes. It evaluates methods like PiCoDeS, Spectral Hashing, and LSH, demonstrating that PiCoDeS achieves the highest keyword accuracy (77.7% hit rate), significantly reducing manual labeling effort while maintaining high-quality video description without relying on bag-of-words features.

ABSTRACT

Automatic video keyword generation is one of the key ingredients in reducing the burden of security officers in analyzing surveillance videos. Keywords or attributes are generally chosen manually based on expert knowledge of surveillance. Most existing works primarily aim at either supervised learning approaches relying on extensive manual labelling or hierarchical probabilistic models that assume the features are extracted using the bag-of-words approach; thus limiting the utilization of the other features. To address this, we turn our attention to automatic attribute discovery approaches. However, it is not clear which automatic discovery approach can discover the most meaningful attributes. Furthermore, little research has been done on how to compare and choose the best automatic attribute discovery methods. In this paper, we propose a novel approach, based on the shared structure exhibited amongst meaningful attributes, that enables us to compare between different automatic attribute discovery approaches.We then validate our approach by comparing various attribute discovery methods such as PiCoDeS on two attribute datasets. The evaluation shows that our approach is able to select the automatic discovery approach that discovers the most meaningful attributes. We then employ the best discovery approach to generate keywords for videos recorded from a surveillance system. This work shows it is possible to massively reduce the amount of manual work in generating video keywords without limiting ourselves to a particular video feature descriptor.

Motivation & Objective

• To address the lack of systematic comparison methods for automatic attribute discovery in surveillance video keyword generation.
• To reduce reliance on extensive manual labeling, which limits scalability and adaptability to unseen events.
• To develop a selection framework that identifies the most meaningful attributes across different discovery methods.
• To validate the framework on real surveillance video datasets and demonstrate its effectiveness in generating accurate, human-interpretable keywords.

Proposed method

• The method uses a distance function based on shared structure in meaningful attributes to measure the meaningfulness of discovered attributes.
• It treats a set of known, meaningful attributes as a 'yardstick' to evaluate and compare different automatic attribute discovery methods.
• The approach is applied to compare PiCoDeS, Spectral Hashing (SH), and Locality-Sensitive Hashing (LSH) on two attribute datasets.
• It evaluates performance using precision (hit rate) of correctly identified keywords per video, with results aggregated per attribute and action.
• The framework enables selection of the best-performing attribute discovery method without requiring additional manual labeling.
• It supports end-to-end keyword generation for surveillance videos using the selected method, minimizing manual effort.

Experimental results

Research questions

• RQ1 Which automatic attribute discovery method produces the most semantically meaningful attributes for surveillance video keyword generation?
• RQ2 How can we objectively compare different attribute discovery methods without relying on manual annotations?
• RQ3 Can a shared structural pattern among meaningful attributes be used to define a reliable metric for attribute quality?
• RQ4 Does the proposed selection method consistently identify the best-performing method across different datasets and video content?
• RQ5 To what extent can the selected method reduce manual labeling effort while maintaining high keyword accuracy?

Key findings

• PiCoDeS achieved the highest keyword hit rate at 77.7%, significantly outperforming Spectral Hashing (55.9%) and LSH (48.3%) on the tested dataset.
• Most attributes discovered by PiCoDeS had a hit rate above 70%, with two attributes achieving 100% accuracy.
• The method reduced the number of required human labeling tasks (HITs) from N (number of training samples) to J (number of discovered attributes), enabling substantial time and cost savings.
• The evaluation confirmed that the proposed distance function effectively identifies the most meaningful attribute sets, validating its use as a selection criterion.
• The framework enables scalable, low-cost video keyword generation without being constrained by specific video feature descriptors.
• The results demonstrate that automatic attribute discovery can be effectively evaluated and selected using structural patterns in meaningful attributes, enabling practical deployment in surveillance systems.