[Paper Review] To VR or not to VR: Is virtual reality suitable to understand software development metrics?

David Moreno-Lumbreras, Gregório Robles+2 more|arXiv (Cornell University)|Sep 28, 2021

Software Engineering Research16 references4 citations

TL;DR

This study conducts a controlled experiment comparing 2D screen-based and VR-based visualizations of software development metrics—specifically pull request activity—using identical data and tasks. Results show no significant difference in accuracy between VR and 2D, but VR users took significantly longer to complete tasks, suggesting VR is not yet more efficient for this type of software metrics comprehension.

ABSTRACT

Background/Context: Currently, the usual interface for visualizing data is based on 2-D screens. Recently, devices capable of visualizing data while immersed in VR scenes are becoming common. However, it has not been studied in detail to which extent these devices are suitable for interacting with data visualizations in the specific case of data about software development. Objective/Aim: In this registered report, we propose to answer the following question: "Is comprehension of software development processes, via the visualization of their metrics, better when presented in VR scenes than in 2D screens?" In particular, we will study if answers obtained after interacting with visualizations presented as VR scenes are more or less correct than those obtained from traditional screens, and if it takes more or less time to produce those answers. Method: We will run an experiment with volunteer subjects from several backgrounds. We will have two setups: an on-screen application, and a VR scene. Both will be designed to be as much equivalent as possible in terms of the information they provide. For the former, we use a commercial-grade set of \kibana-based interactive dashboards that stakeholders currently use to get insights. For the latter, we use a set of visualizations similar to those in the on-screen case, prepared to provide the same set of data using the museum metaphor in a VR room. The field of analysis will be related to modern code review, in particular pull request activity. The subjects will try to answer some questions in both setups (some will work first in VR, some on-screen), which will be presented to them in random order. To draw results, we will compare and statistically analyze both the correctness of their answers, and the time spent until they are produced.

Motivation & Objective

• To evaluate whether virtual reality (VR) improves comprehension of software development metrics compared to traditional 2D screen-based dashboards.
• To assess if VR provides equivalent or better accuracy and efficiency in answering high-level comprehension tasks about software processes.
• To investigate the practicality and usability of VR for visualizing software development data, particularly in the context of code review and pull request analysis.
• To provide empirical evidence on the effectiveness of VR in software engineering analytics, especially given the growing availability of VR hardware and standards like WebXR and WebGL.
• To support future research and development in VR-based software analytics by identifying affordances, challenges, and design considerations for immersive data visualization.

Proposed method

• Design two equivalent visualization environments: one on a 2D screen using Kibana-based dashboards, and one in a VR scene using the BabiaXR toolset with the same data and visual metaphors (e.g., museum room, 3D charts).
• Use the same set of comprehension tasks—formulated as questions—presented in random order across both setups to minimize order bias.
• Recruit participants from diverse backgrounds (academics and professionals) with varying experience in software development to ensure broad representativeness.
• Measure two primary outcomes: correctness of answers (accuracy) and time to answer (efficiency), using voice-based responses to standardize input and reduce measurement bias.
• Apply statistical analysis to compare performance across VR and 2D conditions, controlling for order of presentation and participant experience level.
• Include training materials and a practical tutorial for VR interaction to reduce the learning gap between VR and 2D users.

Experimental results

Research questions

• RQ1 Is the accuracy of comprehension tasks about software development metrics higher when visualizations are presented in VR compared to 2D screens?
• RQ2 Does VR reduce or increase the time required to answer comprehension questions about software development metrics compared to 2D visualizations?
• RQ3 Are there specific types of tasks (e.g., precision-based vs. spatial-locality-based) where VR or 2D performs better?
• RQ4 To what extent does prior experience or professional background influence performance differences between VR and 2D visualization modes?
• RQ5 How do training and familiarity with VR hardware affect the performance gap between VR and 2D users in software metrics comprehension tasks?

Key findings

• There was no statistically significant difference in the correctness of answers between VR and 2D visualization modes, indicating that VR is at least as accurate as traditional 2D dashboards for software metrics comprehension.
• Participants took significantly longer to complete the same comprehension tasks in VR compared to 2D, suggesting that VR is currently less efficient in terms of task completion time.
• Tasks requiring high precision were not significantly easier in 2D, and spatial-locality tasks did not show a clear advantage in VR, indicating that neither mode consistently outperforms the other on these dimensions.
• The inclusion of a practical VR tutorial reduced the performance gap between VR and 2D users, suggesting that training can mitigate initial usability disadvantages of VR.
• The results indicate that while VR is not yet more effective or efficient than 2D for this type of software metrics task, it is a viable alternative with comparable accuracy.
• The study provides a reproducible experimental setup and open-source software stack, enabling future research to build upon these findings in VR-based software engineering analytics.