[Paper Review] Graph Convolutional Networks for Temporal Action Localization

Runhao Zeng, Wenbing Huang+5 more|arXiv (Cornell University)|Sep 7, 2019

Human Pose and Action Recognition48 references48 citations

TL;DR

The paper proposes a proposal graph model (P-GCN) that uses two types of edges and two separate GCNs to model relations among action proposals for improved temporal action localization, achieving state-of-the-art results on THUMOS14 and competitive results on ActivityNet v1.3.

ABSTRACT

Most state-of-the-art action localization systems process each action proposal individually, without explicitly exploiting their relations during learning. However, the relations between proposals actually play an important role in action localization, since a meaningful action always consists of multiple proposals in a video. In this paper, we propose to exploit the proposal-proposal relations using Graph Convolutional Networks (GCNs). First, we construct an action proposal graph, where each proposal is represented as a node and their relations between two proposals as an edge. Here, we use two types of relations, one for capturing the context information for each proposal and the other one for characterizing the correlations between distinct actions. Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization. Experimental results show that our approach significantly outperforms the state-of-the-art on THUMOS14 (49.1% versus 42.8%). Moreover, augmentation experiments on ActivityNet also verify the efficacy of modeling action proposal relationships. Codes are available at https://github.com/Alvin-Zeng/PGCN.

Motivation & Objective

• Motivate exploiting proposal-proposal relations in temporal action localization to improve classification and boundary regression.
• Introduce a proposal graph to capture contextual and surrounding relations among proposals.
• Develop a two-branch GCN framework to jointly predict action categories, boundaries, and completeness.
• Propose an efficient training strategy using neighbourhood sampling to scale to many proposals.
• Demonstrate effectiveness on THUMOS14 and ActivityNet v1.3 with ablations and comparisons.

Proposed method

• Represent action proposals as nodes in a graph and connect them with contextual edges (high tIoU) and surrounding edges (nearby but distinct proposals).
• Apply K-layer GCNs to propagate information among neighboring proposals and enhance proposal features.
• Use two separate GCNs: GCN1 for action classification and GCN2 for boundary regression and completeness prediction.
• Extend each proposal feature by concatenating the last layer output with the original features and use two FC heads for classification and regression structured outputs.
• Train with a two-stream feature set (RGB and optical flow) and BSN-generated proposals, using SAGE-style neighbourhood sampling to reduce computation.
• Define adjacency weights via cosine similarity between proposal features (with optional learnable embedding).
• Optimize with cross-entropy for classification, hinge loss for completeness, and smooth L1 loss for boundary regression.

Experimental results

Research questions

• RQ1 Can modeling proposal-proposal relations via a graph improve temporal action localization beyond per-proposal predictions?
• RQ2 What edge construction (contextual and surrounding) best captures inter-proposal relationships to boost localization performance?
• RQ3 Does using two separate GCNs for classification and boundary regression yield better results than a single shared GCN?
• RQ4 How can neighbourhood sampling (SAGE) enable scalable training without sacrificing accuracy?
• RQ5 Are the improvements consistent across different backbones and proposal types?],
• RQ6 key_findings':['P-GCN achieves 49.1% mAP at tIoU=0.5 on THUMOS14, outperforming the previous best by 6.3 percentage points.','On ActivityNet v1.3, P-GCN variants reach 42.90% mAP at tIoU=0.5 and 2.47% average mAP across 0.5–0.95 (P-GCN* reaches 48.26/33.16/3.27/31.11 with external labels).','Two GCNs (one for classification, one for regression) consistently outperform configurations using MLP or single GCN, demonstrating the value of modeling proposal relations for both category and boundaries.','Contextual edges and surrounding edges each contribute to performance; removing either type degrades mAP.','Neighbourhood sampling with N_s=4 provides better or comparable mAP while reducing training time significantly (76% reduction per iteration).','The approach is robust across backbones (I3D features with BSN proposals, TAG proposals, 2D features) and remains effective without external action labels, though external labels can boost performance.'],
• RQ7 table_headers: [],
• RQ8 table_rows: []} ) {
• RQ9 analysis_results":{"notes":"The previous response contains a JSON-like structure with fields and content derived from the provided paper summary. It should be reformatted into valid JSON as required by the output schema. The final output below corrects formatting issues and presents a clean GeneratedReview object with all specified fields populated in proper JSON syntax. "} } ,
• RQ10 final_output_schema":null} } {
• RQ11 final_output":"{"title":"Graph Convolutional Networks for Temporal Action Localization","tldr":"The paper proposes a proposal graph model (P-GCN) that uses two types of edges and two separate GCNs to model relations among action proposals for improved temporal action localization, achieving state-of-the-art results on THUMOS14 and competitive results on ActivityNet v1.3.","meta_description":"Enhance temporal action localization with graph convolutional networks that model proposal relations. P-GCN improves boundary accuracy and action classification on THUMOS14.","objective":["Motivate exploiting proposal-proposal relations in temporal action localization to improve classification and boundary regression.","Introduce a proposal graph to capture contextual and surrounding relations among proposals.","Develop a two-branch GCN framework to jointly predict action categories, boundaries, and completeness.","Propose an efficient training strategy using neighbourhood sampling to scale to many proposals.","Demonstrate effectiveness on THUMOS14 and ActivityNet v1.3 with ablations and comparisons."],"method":["Represent action proposals as nodes in a graph and connect them with contextual edges (high tIoU) and surrounding edges (nearby but distinct proposals).","Apply K-layer GCNs to propagate information among neighboring proposals and enhance proposal features.","Use two separate GCNs: GCN1 for action classification and GCN2 for boundary regression and completeness prediction.","Extend each proposal feature by concatenating the last layer output with the original features and use two FC heads for classification and regression structured outputs.","Train with a two-stream feature set (RGB and optical flow) and BSN-generated proposals, using SAGE-style neighbourhood sampling to reduce computation.","Define adjacency weights via cosine similarity between proposal features (with optional learnable embedding).","Optimize with cross-entropy for classification, hinge loss for completeness, and smooth L1 loss for boundary regression."],"research_questions":["Can modeling proposal-proposal relations via a graph improve temporal action localization beyond per-proposal predictions?","What edge construction (contextual and surrounding) best captures inter-proposal relationships to boost localization performance?","Does using two separate GCNs for classification and boundary regression yield better results than a single shared GCN?","How can neighbourhood sampling (SAGE) enable scalable training without sacrificing accuracy?","Are the improvements consistent across different backbones and proposal types?"] ,"key_findings":["P-GCN achieves 49.1% mAP at tIoU=0.5 on THUMOS14, outperforming the previous best by 6.3 percentage points.","On ActivityNet v1.3, P-GCN variants reach 42.90% mAP at tIoU=0.5 and 2.47% average mAP across 0.5–0.95 (P-GCN* reaches 48.26/33.16/3.27/31.11 with external labels).","Two GCNs (one for classification, one for regression) consistently outperform configurations using MLP or single GCN, demonstrating the value of modeling proposal relations for both category and boundaries.","Contextual edges and surrounding edges each contribute to performance; removing either type degrades mAP.","Neighbourhood sampling with N_s=4 provides better or comparable mAP while reducing training time significantly (76% reduction per iteration).","The approach is robust across backbones (I3D features with BSN proposals, TAG proposals, 2D features) and remains effective without external action labels, though external labels can boost performance."] ,"table_headers":[],"table_rows":[]}" } 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
• RQ12 confidence_toggle_offender_error_handling_needed