[Paper Review] Region-of-Interest Based Neural Video Compression

Yura Perugachi-Diaz, Guillaume Sautière+4 more|arXiv (Cornell University)|Mar 3, 2022

Image and Signal Denoising Methods4 citations

TL;DR

This paper proposes two novel neural video compression frameworks that prioritize Region-of-Interest (ROI) quality by leveraging ROI masks to adaptively allocate bits. The implicit model de-emphasizes background distortion during training, while the explicit latent-scaling model uses a learned gain tensor to control quantization binwidth per spatial region. Both methods achieve significant rate-distortion gains in ROI regions without requiring pixel-level ROI annotations during training, outperforming existing baselines on DAVIS and generalizing across datasets.

ABSTRACT

Humans do not perceive all parts of a scene with the same resolution, but rather focus on few regions of interest (ROIs). Traditional Object-Based codecs take advantage of this biological intuition, and are capable of non-uniform allocation of bits in favor of salient regions, at the expense of increased distortion the remaining areas: such a strategy allows a boost in perceptual quality under low rate constraints. Recently, several neural codecs have been introduced for video compression, yet they operate uniformly over all spatial locations, lacking the capability of ROI-based processing. In this paper, we introduce two models for ROI-based neural video coding. First, we propose an implicit model that is fed with a binary ROI mask and it is trained by de-emphasizing the distortion of the background. Secondly, we design an explicit latent scaling method, that allows control over the quantization binwidth for different spatial regions of latent variables, conditioned on the ROI mask. By extensive experiments, we show that our methods outperform all our baselines in terms of Rate-Distortion (R-D) performance in the ROI. Moreover, they can generalize to different datasets and to any arbitrary ROI at inference time. Finally, they do not require expensive pixel-level annotations during training, as synthetic ROI masks can be used with little to no degradation in performance. To the best of our knowledge, our proposals are the first solutions that integrate ROI-based capabilities into neural video compression models.

Motivation & Objective

• To address the limitation of uniform bit allocation in neural video codecs, which fails to prioritize perceptually important regions like the human fovea.
• To enable non-uniform rate-distortion optimization in neural video compression by integrating ROI awareness into the learning process.
• To design models that generalize across datasets and ROI specifications without requiring ground-truth pixel-level annotations during training.
• To maintain computational efficiency while achieving superior ROI-specific rate-distortion performance compared to existing neural and traditional codecs.

Proposed method

• The implicit ROI model is trained with a modified distortion loss that de-emphasizes background regions via a background penalty hyperparameter γ.
• The explicit latent-scaling model introduces a gain tensor generated by an auxiliary autoencoder from the ROI mask, which modulates the quantization binwidth of latent features per spatial region.
• The gain tensor acts as a continuous, learnable equivalent to discrete masking in scalar quantization, enabling fine-grained control over quantization resolution.
• Both models are built on the Scale Space Flow (SSF) architecture and are compatible with hyperprior-based video compression frameworks.
• The models use straight-through gradient estimation with additive uniform noise for differentiable quantization, enabling end-to-end training.
• Synthetic ROI masks are used during training with minimal performance degradation, enabling annotation-free training.

Experimental results

Research questions

• RQ1 Can neural video codecs achieve superior rate-distortion performance in Region-of-Interest (ROI) regions by learning to prioritize them during training?
• RQ2 How can ROI-based compression be integrated into neural video codecs without requiring expensive pixel-level annotations for ROI masks?
• RQ3 Can the proposed models generalize to different datasets and ROI specifications at inference time without retraining?
• RQ4 What is the computational cost of introducing ROI-aware compression in neural video codecs compared to standard baselines?
• RQ5 How does the explicit latent-scaling mechanism compare to implicit loss-based de-emphasis in terms of ROI quality and rate-distortion trade-offs?

Key findings

• The proposed implicit and explicit ROI-based models achieve a 69.3% BD-rate gain on the DAVIS dataset, significantly outperforming baseline models in ROI-PSNR.
• The models generalize well to different datasets and ROI specifications at inference time, even when trained on synthetic ROI masks.
• Training with synthetic ROI masks yields visual and quantitative performance comparable to training with ground-truth masks, confirming minimal degradation from weak supervision.
• The runtime overhead of the latent-scaling model is moderate, with a 17% drop in I-frame encoding FPS and 24% drop in decoding FPS on 720p inputs compared to the baseline SSF.
• The explicit latent-scaling model provides better control over quantization resolution and achieves higher ROI quality than the implicit model, especially at low bitrates.
• Both models maintain high perceptual quality in ROI regions while distorting the background more, aligning with human visual system characteristics.