[Paper Review] Minimizing the Motion-to-Photon-delay (MPD) in Virtual Reality Systems

Akanksha Dixit, Smruti R. Sarangi|arXiv (Cornell University)|Jan 25, 2023

Image and Video Quality Assessment4 citations

TL;DR

This paper proposes PredATW, a machine learning-based system that predicts GPU-accelerated Asynchronous Time Warp (ATW) latency in virtual reality systems to schedule time warping as late as possible, minimizing motion-to-photon delay (MPD). It achieves a prediction error of 0.77 ms and reduces deadline misses by 73.1% compared to baseline, offering a low-cost software solution without extra hardware.

ABSTRACT

With the advent of low-power ultra-fast hardware and GPUs, virtual reality (VR) has gained a lot of prominence in the last few years and is being used in various areas such as education, entertainment, scientific visualization, and computer-aided design. VR-based applications are highly interactive, and one of the most important performance metrics for these applications is the motion-to-photon-delay (MPD). MPD is the delay from the users head movement to the time at which the image gets updated on the VR screen. Since the human visual system can even detect an error of a few pixels (very spatially sensitive), the MPD should be as small as possible. Popular VR vendors use the GPU-accelerated Asynchronous Time Warp (ATW) algorithm to reduce the MPD. ATW reduces the MPD if and only if the warping operation finishes just before the display refreshes. However, due to the competition between applications for the shared GPU, the GPU-accelerated ATW algorithm suffers from an unpredictable ATW latency, making it challenging to find the ideal time instance for starting the time warp and ensuring that it completes with the least amount of lag relative to the screen refresh. Hence, the state-of-the-art is to use a separate hardware unit for the time warping operation. Our approach, PredATW, uses an ML-based predictor to predict the ATW latency for a VR application, and then schedule it as late as possible. This is the first work to do so. Our predictor achieves an error of 0.77 ms across several popular VR applications for predicting the ATW latency. As compared to the baseline architecture, we reduce deadline misses by 73.1%.

Motivation & Objective

• Address the challenge of unpredictable ATW latency in GPU-accelerated VR systems, which degrades MPD and user experience.
• Reduce motion-to-photon delay (MPD) in VR applications, which is critical for preventing motion sickness and ensuring immersive interaction.
• Develop a low-cost, software-only solution that avoids the need for dedicated hardware units or complex motion prediction by focusing on ATW latency prediction.
• Enable precise scheduling of time warping operations to complete just before display refresh, minimizing lag and deadline misses.

Proposed method

• Train a machine learning predictor to estimate the latency of the GPU-accelerated Asynchronous Time Warp (ATW) operation based on runtime characteristics.
• Use the predicted ATW latency to determine the optimal time to initiate the time warping kernel, scheduling it as late as possible while ensuring completion before the display refresh.
• Leverage real-time GPU workload data, including concurrent application interference, to improve prediction robustness under noisy conditions.
• Integrate the predictor into the VR rendering pipeline to dynamically adjust ATW launch timing without modifying existing GPU rendering or warping kernels.
• Ensure the system remains compatible with existing VR workloads by operating as a lightweight software layer on top of standard GPU-accelerated ATW.
• Use a regression model to predict ATW latency with minimal overhead, achieving high accuracy even under shared GPU resource contention.

Experimental results

Research questions

• RQ1 Can a machine learning model accurately predict the variable latency of GPU-accelerated ATW operations in real time?
• RQ2 How does predicting ATW latency improve MPD reduction compared to fixed or heuristic scheduling strategies?
• RQ3 To what extent can a software-only solution reduce deadline misses in ATW scheduling without dedicated hardware?
• RQ4 What is the achievable prediction accuracy of ATW latency under real-world GPU workload interference?
• RQ5 Can the proposed approach significantly reduce MPD while maintaining compatibility with existing VR rendering stacks?

Key findings

• The proposed ML-based predictor achieves a mean absolute error (MAE) of only 0.77 ms in predicting ATW latency across multiple popular VR applications.
• PredATW reduces deadline misses in ATW scheduling by 73.1% compared to the baseline GPU-accelerated ATW approach.
• The system maintains high accuracy even under GPU resource contention from other applications, demonstrating robustness in shared environments.
• The solution achieves significant MPD reduction without requiring additional hardware, such as dedicated warping units or dual-GPU architectures.
• The approach enables optimal scheduling of time warping to complete just before display refresh, minimizing motion-to-photon delay.
• The results suggest that future improvements using ensemble learning or more training data could reduce prediction error below 0.5 ms.