[Paper Review] You Need to Read Again: Multi-granularity Perception Network for Moment Retrieval in Videos

Xin Sun, Xuan Wang+3 more|arXiv (Cornell University)|May 25, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Multi-granularity Perception Network (MGPN), a novel framework for moment retrieval in videos that emulates human reading strategies by performing coarse-to-fine perception of video and language features. By integrating a coarse-grained encoder with a fine-grained feature encoder and conditioned interaction module, MGPN improves alignment accuracy on Charades-STA, TACoS, and ActivityNet Captions, outperforming prior state-of-the-art methods with reduced computational cost through a lightweight choice comparison module.

ABSTRACT

Moment retrieval in videos is a challenging task that aims to retrieve the most relevant video moment in an untrimmed video given a sentence description. Previous methods tend to perform self-modal learning and cross-modal interaction in a coarse manner, which neglect fine-grained clues contained in video content, query context, and their alignment. To this end, we propose a novel Multi-Granularity Perception Network (MGPN) that perceives intra-modality and inter-modality information at a multi-granularity level. Specifically, we formulate moment retrieval as a multi-choice reading comprehension task and integrate human reading strategies into our framework. A coarse-grained feature encoder and a co-attention mechanism are utilized to obtain a preliminary perception of intra-modality and inter-modality information. Then a fine-grained feature encoder and a conditioned interaction module are introduced to enhance the initial perception inspired by how humans address reading comprehension problems. Moreover, to alleviate the huge computation burden of some existing methods, we further design an efficient choice comparison module and reduce the hidden size with imperceptible quality loss. Extensive experiments on Charades-STA, TACoS, and ActivityNet Captions datasets demonstrate that our solution outperforms existing state-of-the-art methods. Codes are available at github.com/Huntersxsx/MGPN.

Motivation & Objective

• To address the limitation of existing two-stage moment retrieval models that perform coarse-grained intra- and inter-modal perception, neglecting fine-grained semantic details.
• To improve alignment accuracy by modeling moment retrieval as a multi-choice reading comprehension task, inspired by human rereading and deeper reasoning strategies.
• To reduce computational overhead without sacrificing performance by designing a lightweight choice comparison module and optimizing hidden size.
• To enable plug-and-play integration of fine-grained perception into existing two-stage models for broader applicability.

Proposed method

• Formulates moment retrieval as a multi-choice reading comprehension task, emulating human strategies such as rereading and enhanced alignment.
• Employs a coarse-grained feature encoder and co-attention mechanism to capture initial intra- and inter-modal representations.
• Introduces a fine-grained feature encoder and a conditioned interaction module to refine representations through deeper reasoning, mimicking rereading.
• Designs an efficient choice comparison module that reduces hidden size with minimal performance loss, enhancing inference efficiency.
• Uses a multi-modal co-attention module to improve video-query alignment in the fine-grained perception stage.
• Applies a stacked convolution block architecture in the fine-grained encoder, achieving better performance and smaller model size than non-local or standard convolutional blocks.

Experimental results

Research questions

• RQ1 Can modeling moment retrieval as a multi-choice reading comprehension task improve alignment accuracy by emulating human rereading behavior?
• RQ2 Does multi-granularity perception—coarse-to-fine processing of intra- and inter-modal features—lead to better performance on moment retrieval benchmarks?
• RQ3 Can a lightweight choice comparison module reduce model size and inference cost without degrading performance?
• RQ4 To what extent do fine-grained self-modal encoding and cross-modal alignment improve existing two-stage models when applied as plug-in modules?

Key findings

• MGPN achieves state-of-the-art performance on Charades-STA, TACoS, and ActivityNet Captions, with R@1,0.5 of 47.92% on ActivityNet Captions.
• The MGPN 256 variant achieves superior performance with only 256 hidden units, reducing model size by 4.8× compared to 2D-TAN while improving performance by 6.4%.
• The plug-and-play experiments show that adding fine-grained perception to 2D-TAN and RaNet boosts R@1,0.5 by 5.77% and 0.87%, respectively, demonstrating generalization and effectiveness.
• Experiments confirm that a hidden size of 256 strikes the best balance between performance and efficiency, as larger sizes (e.g., 512) yield only marginal gains at exponential cost.
• The fine-grained perception module significantly improves retrieval precision on visually similar moments, as shown in qualitative results on ActivityNet Captions and TACoS.
• The proposed multi-modal co-attention module enhances cross-modal alignment, especially for subtle cues like 'left' hand, which are often missed in coarse-grained models.