[Paper Review] BMN: Boundary-Matching Network for Temporal Action Proposal Generation

Tianwei Lin, Xiao Liu+3 more|arXiv (Cornell University)|Jul 23, 2019

Human Pose and Action Recognition35 references73 citations

TL;DR

BMN introduces a Boundary-Matching mechanism to evaluate confidence for densely distributed temporal proposals, enabling end-to-end generation of precise proposals and reliable scores, improving proposal generation and detection performance.

ABSTRACT

Temporal action proposal generation is an challenging and promising task which aims to locate temporal regions in real-world videos where action or event may occur. Current bottom-up proposal generation methods can generate proposals with precise boundary, but cannot efficiently generate adequately reliable confidence scores for retrieving proposals. To address these difficulties, we introduce the Boundary-Matching (BM) mechanism to evaluate confidence scores of densely distributed proposals, which denote a proposal as a matching pair of starting and ending boundaries and combine all densely distributed BM pairs into the BM confidence map. Based on BM mechanism, we propose an effective, efficient and end-to-end proposal generation method, named Boundary-Matching Network (BMN), which generates proposals with precise temporal boundaries as well as reliable confidence scores simultaneously. The two-branches of BMN are jointly trained in an unified framework. We conduct experiments on two challenging datasets: THUMOS-14 and ActivityNet-1.3, where BMN shows significant performance improvement with remarkable efficiency and generalizability. Further, combining with existing action classifier, BMN can achieve state-of-the-art temporal action detection performance.

Motivation & Objective

• Motivate improved temporal action proposal generation with precise boundaries and reliable confidence scores.
• Introduce a Boundary-Matching mechanism to evaluate all dense proposals simultaneously.
• Develop an end-to-end BMN framework that jointly learns boundary probabilities and BM confidence maps.
• Demonstrate efficiency and effectiveness on THUMOS-14 and ActivityNet-1.3 datasets.

Proposed method

• Define a proposal as a boundary-start and boundary-end pair and construct a 2D BM confidence map over all such pairs.
• Propose BM layer to sample features between start and end boundaries, producing BM feature maps that encode context for all proposals.
• Use a two-branch BMN architecture: Temporal Evaluation Module outputs boundary probability sequences; Proposal Evaluation Module outputs BM confidence maps.
• Train TEM and PEM jointly in a unified multi-task framework with TEM losses for boundaries and PEM losses for BM confidence maps.
• Fuse boundary probabilities with BM confidences to form final proposal scores and apply Soft-NMS for suppression.

Experimental results

Research questions

• RQ1 Can a boundary-matching mechanism provide reliable confidence scores for densely distributed, flexible-duration proposals?
• RQ2 Does an end-to-end BMN model yield superior proposal quality and efficiency compared to prior bottom-up/top-down methods?
• RQ3 How well does BMN generalize to unseen action categories and datasets?
• RQ4 What is the impact of joint training of TEM and PEM on proposal generation and detection performance?

Key findings

• BMN achieves significantly better proposal generation metrics than state-of-the-art methods on ActivityNet-1.3 and THUMOS-14.
• On ActivityNet-1.3 validation, BMN improves AUC to 67.10% and AR@100 to 75.01 with TEM+PEM in the ablation study.
• BMN shows strong generalizability, with small performance drops on unseen action subsets in ActivityNet-1.3.
• Jointly training TEM and PEM yields higher recall and faster inference than separately trained configurations.
• BMN-based proposals combined with action classifiers yield state-of-the-art temporal action detection performance on ActivityNet-1.3 and competitive results on THUMOS-14.