[Paper Review] NIRVANA: Neural Implicit Representations of Videos with Adaptive Networks and Autoregressive Patch-wise Modeling

Shishira R Maiya, Sharath Girish+7 more|arXiv (Cornell University)|Dec 30, 2022

Advanced Image Processing Techniques4 citations

TL;DR

NIRVANA proposes a scalable, autoregressive neural implicit video representation that models videos through patch-wise prediction across temporal clips, initializing each clip's network from the prior using adaptive, quantized weights. It achieves 12× faster encoding and variable bitrate compression while maintaining high reconstruction quality, outperforming prior methods on UVG at 37.70 PSNR with lower BPP.

ABSTRACT

Implicit Neural Representations (INR) have recently shown to be powerful tool for high-quality video compression. However, existing works are limiting as they do not explicitly exploit the temporal redundancy in videos, leading to a long encoding time. Additionally, these methods have fixed architectures which do not scale to longer videos or higher resolutions. To address these issues, we propose NIRVANA, which treats videos as groups of frames and fits separate networks to each group performing patch-wise prediction. This design shares computation within each group, in the spatial and temporal dimensions, resulting in reduced encoding time of the video. The video representation is modeled autoregressively, with networks fit on a current group initialized using weights from the previous group's model. To further enhance efficiency, we perform quantization of the network parameters during training, requiring no post-hoc pruning or quantization. When compared with previous works on the benchmark UVG dataset, NIRVANA improves encoding quality from 37.36 to 37.70 (in terms of PSNR) and the encoding speed by 12X, while maintaining the same compression rate. In contrast to prior video INR works which struggle with larger resolution and longer videos, we show that our algorithm is highly flexible and scales naturally due to its patch-wise and autoregressive designs. Moreover, our method achieves variable bitrate compression by adapting to videos with varying inter-frame motion. NIRVANA achieves 6X decoding speed and scales well with more GPUs, making it practical for various deployment scenarios.

Motivation & Objective

• To address the inefficiency and lack of scalability in existing implicit neural representation (INR) methods for video compression, which suffer from long encoding times and fixed architectures.
• To exploit spatio-temporal redundancies in videos by modeling patches across groups of frames rather than pixels or full frames.
• To enable flexible compression for videos of varying resolution, duration, and motion complexity through adaptive, autoregressive network initialization and quantization.
• To achieve high compression efficiency without post-hoc pruning or quantization by integrating quantization during training.
• To improve decoding speed and scalability across GPUs, making INR-based video compression practical for real-world deployment.

Proposed method

• The method segments videos into temporal clips (groups of frames) and fits a separate neural network to each clip for patch-wise prediction of spatio-temporal volumes.
• Each network is initialized using the trained weights from the previous clip’s network, enabling autoregressive modeling that exploits temporal coherence.
• The framework employs quantization of network weights during training, eliminating the need for post-hoc compression techniques and reducing model entropy.
• Residuals between consecutive clip networks are encoded to further compress the model, with entropy regularization controlling the PSNR-BPP tradeoff.
• The architecture is resolution-agnostic: patch coordinates are used as input, allowing adaptation to different spatial resolutions without architectural changes.
• Training is performed with a loss function that includes a regularization term on the entropy of the quantized weights to minimize bit rate.

Experimental results

Research questions

• RQ1 Can an autoregressive, patch-wise INR framework improve encoding speed and compression efficiency in video representation compared to prior pixel- or frame-wise methods?
• RQ2 How does the autoregressive initialization of clip-specific networks affect model performance and scalability across varying video durations and resolutions?
• RQ3 To what extent can in-network quantization during training eliminate the need for post-hoc model compression while maintaining high reconstruction quality?
• RQ4 How does the method adapt to varying inter-frame motion, and can it achieve variable bitrate compression naturally?
• RQ5 What is the impact of patch size, group size, and training iterations on the PSNR-BPP tradeoff in video INR compression?

Key findings

• NIRVANA achieves a PSNR of 37.70 on the UVG dataset, a 0.34 improvement over the prior state-of-the-art (37.36), while maintaining the same compression rate.
• The method reduces encoding time by 12× compared to previous INR-based video compression methods, significantly improving practicality.
• Decoding speed is accelerated by 6×, and the framework scales efficiently with more GPUs, enabling faster inference in distributed settings.
• Increasing patch size from 8×8 to 48×48 improves PSNR but increases BPP due to higher residual entropy in dynamic scenes.
• A frame group size of 3 achieves the best PSNR-BPP tradeoff in the sub-0.8 BPP regime, with larger groups degrading performance on dynamic content.
• Longer training iterations (up to 5000) yield diminishing but consistent PSNR improvements, indicating that extended training enhances reconstruction quality without increasing bit rate.