[Paper Review] Privacy-Preserving Deep Action Recognition: An Adversarial Learning Framework and A New Dataset

Zhenyu Wu, Haotao Wang+3 more|arXiv (Cornell University)|Jun 12, 2019

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes a novel adversarial learning framework for privacy-preserving video action recognition that jointly optimizes utility (action recognition) and privacy (anonymization) by learning a transformation to obscure sensitive attributes. It introduces model restarting and ensemble strategies to enhance universal privacy protection and releases the first benchmark dataset, PA-HMDB51, with per-frame annotations for actions and privacy attributes (face, gender, skin color, nudity, relationships), achieving strong privacy-utility trade-offs in experiments.

ABSTRACT

We investigate privacy-preserving, video-based action recognition in deep learning, a problem with growing importance in smart camera applications. A novel adversarial training framework is formulated to learn an anonymization transform for input videos such that the trade-off between target utility task performance and the associated privacy budgets is explicitly optimized on the anonymized videos. Notably, the privacy budget, often defined and measured in task-driven contexts, cannot be reliably indicated using any single model performance because strong protection of privacy should sustain against any malicious model that tries to steal private information. To tackle this problem, we propose two new optimization strategies of model restarting and model ensemble to achieve stronger universal privacy protection against any attacker models. Extensive experiments have been carried out and analyzed. On the other hand, given few public datasets available with both utility and privacy labels, the data-driven (supervised) learning cannot exert its full power on this task. We first discuss an innovative heuristic of cross-dataset training and evaluation, enabling the use of multiple single-task datasets (one with target task labels and the other with privacy labels) in our problem. To further address this dataset challenge, we have constructed a new dataset, termed PA-HMDB51, with both target task labels (action) and selected privacy attributes (skin color, face, gender, nudity, and relationship) annotated on a per-frame basis. This first-of-its-kind video dataset and evaluation protocol can greatly facilitate visual privacy research and open up other opportunities. Our codes, models, and the PA-HMDB51 dataset are available at https://github.com/VITA-Group/PA-HMDB51.

Motivation & Objective

• To address the growing challenge of visual privacy in smart camera applications where raw video data may expose sensitive attributes like faces and gender.
• To develop a deep learning framework that learns an anonymization transform to preserve action recognition utility while minimizing privacy leakage.
• To overcome the 'universal privacy protection' challenge by designing strategies that defend against any unseen attacker model.
• To address the lack of public datasets with both utility (action) and privacy labels by constructing PA-HMDB51, a new benchmark dataset with per-frame annotations.
• To evaluate the framework using both automated metrics and human studies to validate privacy protection and utility preservation.

Proposed method

• Formulates an adversarial training framework that jointly optimizes action recognition accuracy and privacy budget performance via a differentiable anonymization transform.
• Employs three optimization strategies—beam search, entropy-based, and model restarting—to stabilize training and improve performance.
• Introduces model restarting and ensemble techniques to enhance the generalization of the anonymization model against diverse attacker models, addressing the '∀ challenge' of universal privacy.
• Uses a cross-dataset training and evaluation protocol to leverage existing datasets with utility or privacy labels, reducing domain gap issues.
• Constructs PA-HMDB51, a new video dataset with per-frame annotations for 51 actions and 5 privacy attributes (face, gender, skin color, nudity, relationship), enabling end-to-end evaluation.
• Employs a multi-task loss combining utility (action recognition) and privacy (attribute prediction) objectives, with adaptive thresholds for privacy suppression.

Experimental results

Research questions

• RQ1 Can an end-to-end deep learning framework effectively balance action recognition performance and privacy protection in video data?
• RQ2 How can universal privacy protection be achieved against any potential attacker model, rather than just a specific one?
• RQ3 To what extent can model restarting and ensemble strategies improve the robustness of anonymization against unseen attacker models?
• RQ4 How does the proposed framework perform when evaluated on a new, unified dataset with both action and privacy annotations?
• RQ5 Can human evaluators reliably distinguish actions while failing to identify private attributes in anonymized videos?

Key findings

• The proposed framework achieves a ~8% reduction in privacy cMAP (comprehensive mean average precision) when M=4, with minimal degradation in action recognition accuracy, demonstrating effective privacy-utility trade-offs.
• Model restarting and ensemble strategies significantly improve privacy protection, especially against unseen attacker models, outperforming baseline methods in universal privacy defense.
• The human study confirms that actions remain identifiable in anonymized videos, while privacy attributes like face and gender are no longer recognizable to human observers.
• The 'face' attribute is more easily suppressed than 'gender', indicating asymmetric protection costs, with gender being more resilient due to cues in body shape and clothing.
• The PA-HMDB51 dataset enables consistent evaluation of privacy and utility on the same data, reducing domain shift issues present in prior cross-dataset protocols.
• The Ours-Entropy (restarting) method with M=4 achieves the best trade-off, outperforming naive downsampling and other baselines across all evaluation metrics.