[Paper Review] Cross-Modal Transferable Adversarial Attacks from Images to Videos

Zhipeng Wei, Jingjing Chen+2 more|arXiv (Cornell University)|Dec 10, 2021

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes the Image To Video (I2V) attack, a cross-modal adversarial attack that transfers perturbations from ImageNet-pretrained image models to black-box video recognition models by minimizing cosine similarity between intermediate features of adversarial and benign frames. The method achieves 77.88% and 65.68% average attack success rates on Kinetics-400 and UCF-101, respectively, demonstrating significant cross-modal transferability.

ABSTRACT

Recent studies have shown that adversarial examples hand-crafted on one white-box model can be used to attack other black-box models. Such cross-model transferability makes it feasible to perform black-box attacks, which has raised security concerns for real-world DNNs applications. Nevertheless, existing works mostly focus on investigating the adversarial transferability across different deep models that share the same modality of input data. The cross-modal transferability of adversarial perturbation has never been explored. This paper investigates the transferability of adversarial perturbation across different modalities, i.e., leveraging adversarial perturbation generated on white-box image models to attack black-box video models. Specifically, motivated by the observation that the low-level feature space between images and video frames are similar, we propose a simple yet effective cross-modal attack method, named as Image To Video (I2V) attack. I2V generates adversarial frames by minimizing the cosine similarity between features of pre-trained image models from adversarial and benign examples, then combines the generated adversarial frames to perform black-box attacks on video recognition models. Extensive experiments demonstrate that I2V can achieve high attack success rates on different black-box video recognition models. On Kinetics-400 and UCF-101, I2V achieves an average attack success rate of 77.88% and 65.68%, respectively, which sheds light on the feasibility of cross-modal adversarial attacks.

Motivation & Objective

• To investigate cross-modal transferability of adversarial perturbations from image models to video models.
• To address the challenge of transferring adversarial examples across different data modalities (images vs. videos) without access to white-box video models.
• To develop a method that leverages shared intermediate feature space between image and video models for effective black-box video attacks.
• To improve transferability by optimizing feature-level similarity in the intermediate layers of image models.
• To validate the feasibility and effectiveness of cross-modal adversarial attacks in real-world DNN applications.

Proposed method

• The I2V attack generates adversarial video frames by minimizing the cosine similarity between intermediate features of adversarial and benign frames in an ImageNet-pretrained image model.
• Each frame of a video clip is processed independently through the image model to generate adversarial frames using feature-space optimization.
• The method exploits the observed similarity in low-level feature representations between image and video models to enable cross-modal transfer.
• The attack uses a loss function that encourages adversarial features to become orthogonal to benign features in the intermediate layer space.
• An ensemble variant, ENS-I2V, is proposed to further improve transferability by attacking with multiple image models.
• Temporal consistency is preserved by grouping the generated adversarial frames according to the original video clip's temporal order.

Figure 1 : Overview of the proposed I2V attack. Given a video clip with a true label of “Catching or throwing baseball”, where each frame is input into the ImageNet-pretrained image model separately. Then the image model generates adversarial frames by minimizing the cosine similarity between featur

Experimental results

Research questions

• RQ1 Can adversarial perturbations generated on image models successfully transfer to attack black-box video recognition models?
• RQ2 Is there a shared intermediate feature space between image and video models that enables cross-modal adversarial transfer?
• RQ3 How can adversarial attacks be designed for video models when no white-box access to video models is available?
• RQ4 What optimization objective maximizes transferability from image to video models in the absence of video labels?
• RQ5 Can ensemble attacks over multiple image models further improve cross-modal attack success rates?

Key findings

• The I2V attack achieves an average attack success rate of 77.88% on Kinetics-400 and 65.68% on UCF-101, demonstrating strong cross-modal transferability.
• The proposed method outperforms fine-tuned baseline attacks (e.g., ILAF) even though it does not require white-box access to video models.
• On Kinetics-400, the I2V attack performs worse than baselines when the white-box model is TPN-101, likely due to TPN-101’s superior motion modeling.
• Pearson correlation coefficients (PCC) between cosine similarity trends of image and video features are close to 1, indicating strong linear correlation and effective feature-level alignment.
• The ENS-I2V variant further boosts transferability, showing that ensemble attacks enhance robustness across diverse video model architectures.
• The results confirm that cross-modal adversarial transferability is feasible and effective, even across distinct modalities like images and videos.

Figure 2 : Cosine similarity analysis of intermediate features between image models and video models on benign examples and adversarial examples. The cosine similarities are calculated on the averaged intermediate features extracted from 400 randomly selected videos in Kinetics-400. Adversarial vide