[Paper Review] Explainable Event Recognition

Imran Khan, Kashif Ahmad+4 more|arXiv (Cornell University)|Oct 2, 2021

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes an explainable event recognition framework using Grad-CAM and an Xception-based CNN to interpret model decisions in image-based event recognition. It demonstrates that 78–84% of predictions are based on event-salient objects/regions, with F1-scores of 0.91, 0.94, and 0.97 on natural disasters, social, and sports events, respectively, validating the model's interpretability and generalization capabilities.

ABSTRACT

The literature shows outstanding capabilities for CNNs in event recognition in images. However, fewer attempts are made to analyze the potential causes behind the decisions of the models and exploring whether the predictions are based on event-salient objects or regions? To explore this important aspect of event recognition, in this work, we propose an explainable event recognition framework relying on Grad-CAM and an Xception architecture-based CNN model. Experiments are conducted on three large-scale datasets covering a diversified set of natural disasters, social, and sports events. Overall, the model showed outstanding generalization capabilities obtaining overall F1-scores of 0.91, 0.94, and 0.97 on natural disasters, social, and sports events, respectively. Moreover, for subjective analysis of activation maps generated through Grad-CAM for the predicted samples of the model, a crowdsourcing study is conducted to analyze whether the model's predictions are based on event-related objects/regions or not? The results of the study indicate that 78%, 84%, and 78% of the model decisions on natural disasters, sports, and social events datasets, respectively, are based onevent-related objects or regions.

Motivation & Objective

• To address the lack of interpretability in deep learning-based event recognition models, particularly whether predictions rely on event-salient objects or regions.
• To evaluate whether the model's decisions are grounded in semantically relevant visual content by analyzing activation maps.
• To establish a baseline for explainable event recognition across diverse event categories, including natural disasters, social, and sports events.
• To conduct a crowd-sourcing study to validate whether model predictions are based on event-related visual cues.

Proposed method

• The framework employs an Xception architecture pre-trained on ImageNet for feature extraction and event classification.
• Grad-CAM is applied to generate class activation maps from the final convolutional layer, highlighting regions contributing most to the prediction.
• Activation maps are visualized and analyzed to determine if salient regions correspond to event-relevant objects or scenes.
• A crowd-sourcing study labels whether each prediction is based on event-salient regions, using majority voting across three annotators.
• The model is evaluated on three large-scale datasets: natural disasters, social events, and sports events, covering diverse visual and semantic complexity.
• Performance is measured using F1-scores and subjective analysis accuracy to assess both predictive performance and interpretability.

Experimental results

Research questions

• RQ1 Which parts of an image contribute most to the model’s prediction, and are they aligned with event-salient objects or regions?
• RQ2 Do the model’s predictions rely on event-related visual content, or are they influenced by irrelevant image components?
• RQ3 Are the model’s decisions interpretable and semantically meaningful to human observers?
• RQ4 How well does the model generalize across diverse event categories such as natural disasters, social events, and sports events?

Key findings

• The model achieved F1-scores of 0.91, 0.94, and 0.97 on natural disasters, social events, and sports events datasets, respectively, indicating strong generalization.
• Subjective analysis revealed that 78% of predictions on natural disasters, 84% on social events, and 78% on sports events were based on event-salient objects or regions.
• The highest individual class accuracy was 0.98 for hockey in sports events, while the lowest was 0.53 for F1 race, indicating class-specific challenges.
• Misclassifications were often due to visual similarity with confusing classes or absence of clear event-salient objects.
• Despite strong performance, the gap between current accuracy (78–84%) and ideal 100% suggests room for improvement in event-salience modeling.
• Activation maps confirmed that the model primarily attends to event-relevant regions, validating the interpretability of Grad-CAM in this context.