[Paper Review] ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring

Dongxu Li, Chenchen Xu+6 more|arXiv (Cornell University)|Mar 7, 2021

Advanced Image Processing Techniques47 references4 citations

TL;DR

This paper proposes ARVo, a novel video deblurring method that learns all-range volumetric correspondence in feature space using a correlation volume pyramid to handle large displacements from fast motions. By combining implicit feature-based correspondence with progressive generative adversarial training and a new high-frame-rate dataset (HFR-DVD), ARVo achieves state-of-the-art performance on both DVD and HFR-DVD datasets, outperforming prior methods in both quantitative metrics and visual quality.

ABSTRACT

Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions. In order to utilize neighboring sharp patches, typical methods rely mainly on homography or optical flows to spatially align neighboring blurry frames. However, such explicit approaches are less effective in the presence of fast motions with large pixel displacements. In this work, we propose a novel implicit method to learn spatial correspondence among blurry frames in the feature space. To construct distant pixel correspondences, our model builds a correlation volume pyramid among all the pixel-pairs between neighboring frames. To enhance the features of the reference frame, we design a correlative aggregation module that maximizes the pixel-pair correlations with its neighbors based on the volume pyramid. Finally, we feed the aggregated features into a reconstruction module to obtain the restored frame. We design a generative adversarial paradigm to optimize the model progressively. Our proposed method is evaluated on the widely-adopted DVD dataset, along with a newly collected High-Frame-Rate (1000 fps) Dataset for Video Deblurring (HFR-DVD). Quantitative and qualitative experiments show that our model performs favorably on both datasets against previous state-of-the-art methods, confirming the benefit of modeling all-range spatial correspondence for video deblurring.

Motivation & Objective

• To address the limitation of explicit alignment methods like optical flow and homography in handling large displacements due to fast motions in video deblurring.
• To develop an implicit correspondence learning mechanism that captures long-range spatial dependencies across frames without relying on motion estimation.
• To improve restoration quality by maintaining high-resolution features in the reference frame while aggregating context from neighboring frames through a multi-scale correlation volume pyramid.
• To enhance temporal consistency and visual quality through progressive training with a generative adversarial framework.

Proposed method

• Constructs a correlation volume pyramid by matching all pixel-pairs between the reference frame and neighboring frames in the feature space, enabling all-range correspondence learning.
• Designs a correlative aggregation module that enhances the reference frame's features by maximizing correlations with neighboring features across the entire spatial range.
• Maintains the spatial resolution of the reference frame while subsampling only the neighboring frames' features, preserving fine-grained details.
• Employs a progressive training scheme where each stage uses restored frames from the previous stage as input, gradually improving restoration quality.
• Introduces a temporal consistency loss in a generative adversarial training paradigm, where a discriminator distinguishes between real sharp frame sequences and generated ones.
• Uses a multi-scale correlation volume to improve receptive field while retaining high-resolution details, avoiding the limitations of local receptive fields in previous methods.

Figure 2 : Overview of the proposed video deblurring pipeline. Our method first warps the neighboring frame to the reference frame for feature extraction. Then it computes all-range volumetric correspondence between the reference and neighboring frames as a correlation volume pyramid. Next, it relie

Experimental results

Research questions

• RQ1 Can implicit feature-space correspondence learning outperform explicit motion-based alignment in video deblurring under large displacements?
• RQ2 How does modeling all-range pixel-pair correlations across frames improve deblurring performance for fast-motion scenes?
• RQ3 To what extent does progressive training with adversarial supervision enhance temporal consistency and visual quality in video deblurring?
• RQ4 How does the proposed correlation volume pyramid compare to optical flow or homography-based alignment in terms of robustness and accuracy?
• RQ5 What is the impact of using a high-frame-rate dataset (1000 fps) on evaluating and improving video deblurring models?

Key findings

• ARVo achieves a PSNR of 32.80 and SSIM of 0.9352 on the DVD dataset, outperforming previous state-of-the-art methods.
• On the newly introduced HFR-DVD dataset, ARVo achieves superior qualitative results, especially in handling fast-motion blurs with improved visual consistency and detail recovery.
• The ablation study shows that combining optical flow-based alignment and correlation volume alignment yields the best performance (PSNR: 32.80, SSIM: 0.9352), demonstrating their complementary nature.
• Progressive training with two stages improves PSNR by 0.52 dB compared to single-stage training, with diminishing returns beyond two stages.
• The generative adversarial training improves visual consistency across frames, as shown in qualitative comparisons, even though the improvement in PSNR is marginal (-0.01), indicating a perceptual benefit.
• Feature visualization confirms that the correlative aggregation module effectively refines highly blurred regions while preserving less blurred areas, demonstrating targeted feature enhancement.

Figure 3 : Illustration of the correlative aggregation module, which aggregates the features for the reference frame based on the normalized correlation volume.