[Paper Review] Marginalized Average Attentional Network for Weakly-Supervised Learning

Yuan Yuan, Yueming Lyu+3 more|arXiv (Cornell University)|May 21, 2019

Neural Networks and Applications62 citations

TL;DR

MAAN introduces a marginalized average aggregation (MAA) to suppress dominance of the most salient snippets in weakly-supervised temporal action localization, improving the localization of dense action regions. It provides an end-to-end trainable framework with theoretical guarantees and a fast O(T^2) computation.

ABSTRACT

In weakly-supervised temporal action localization, previous works have failed to locate dense and integral regions for each entire action due to the overestimation of the most salient regions. To alleviate this issue, we propose a marginalized average attentional network (MAAN) to suppress the dominant response of the most salient regions in a principled manner. The MAAN employs a novel marginalized average aggregation (MAA) module and learns a set of latent discriminative probabilities in an end-to-end fashion. MAA samples multiple subsets from the video snippet features according to a set of latent discriminative probabilities and takes the expectation over all the averaged subset features. Theoretically, we prove that the MAA module with learned latent discriminative probabilities successfully reduces the difference in responses between the most salient regions and the others. Therefore, MAAN is able to generate better class activation sequences and identify dense and integral action regions in the videos. Moreover, we propose a fast algorithm to reduce the complexity of constructing MAA from O($2^T$) to O($T^2$). Extensive experiments on two large-scale video datasets show that our MAAN achieves superior performance on weakly-supervised temporal action localization

Motivation & Objective

• Motivate improved localization of dense and integral action regions under weak supervision.
• Develop an end-to-end trainable aggregation mechanism that reduces dominance of the most salient snippets.
• Provide theoretical analysis showing latent discriminative probabilities preserve partial order and suppress dominant responses.
• Deliver a fast algorithm to compute marginalized aggregations with polynomial complexity.
• Demonstrate superior performance on THUMOS14 and ActivityNet1.3 datasets.

Proposed method

• Introduce Marginalized Average Aggregation (MAA) which samples subset features via latent probabilities p_t and computes the expectation over all subsets.
• Represent the final aggregation as E[ sum z_i x_i / sum z_i ] where z_i ~ Bernoulli(p_i).
• Prove properties: partial order preservation (p_i preserves order of attention) and dominant-response suppression (latent probabilities reduce gaps in response).
• Derive a fast O(T^2) iterative algorithm to compute the aggregation instead of the naive O(2^T) enumeration.
• Integrate MAA into an MAAN architecture that replaces the STPN aggregators with MAA and latent discriminative probability p_t; train end-to-end with video-level labels using cross-entropy loss.
• Produce class activation sequences (CAS) for temporal proposals via s^c_t = p_t * sigmoid(w_c^T x_t).

Experimental results

Research questions

• RQ1 Can marginalized average aggregation (MAA) suppress dominant responses and encourage dense, integral action regions under weak supervision?
• RQ2 Does MAAN improve temporal action localization accuracy on standard benchmarks compared to existing aggregators?
• RQ3 What is the computational cost of MAA and can it be efficiently trained end-to-end?
• RQ4 Do latent discriminative probabilities retain useful ordering of snippet relevance while reducing overemphasis on a few regions?
• RQ5 How does MAAN perform on THUMOS14 and ActivityNet1.3 datasets under weak supervision?

Key findings

• MAAN consistently outperforms baseline aggregators on THUMOS14 across IoU thresholds.
• MAAN achieves higher AP@IoU and Cls mAP than STPN and other baselines on THUMOS14 (e.g., MAAN AP@IoU 59.8, Cls mAP 50.8 at IoU 0.1).
• Normalized vs. SoftMax normalization analyses show MAAN's advantage stems from marginalization over subsets rather than simple weighting, yielding denser, more integral action segments.
• Theoretical results show latent probabilities preserve partial order with respect to attention and reduce the gap between most salient and other regions.
• A fast iterative O(T^2) algorithm enables practical end-to-end training of MAAN.
• On THUMOS14, MAAN (ours) achieves AP@IoU values of 59.8, 50.8, 41.1, 30.6, 20.3, 12.0, 6.9 across increasing IoU thresholds, with Cls mAP 94.1 (as reported in the table).