[Paper Review] Where's My Drink? Enabling Peripheral Real World Interactions While Using HMDs

Pulkit Budhiraja, Rajinder Singh Sodhi+4 more|arXiv (Cornell University)|Feb 16, 2015

Virtual Reality Applications and Impacts9 references23 citations

TL;DR

This paper proposes a mixed reality rendering technique using stereo cameras to selectively integrate physical world elements—such as hands, objects, and environmental edges—into virtual reality (VR) via head-mounted displays (HMDs), enhancing peripheral interaction without breaking immersion. The Object & Hands with Context (OHC) rendering, which displays the user's hands, target objects, and salient edges of the physical environment, was found to be the most effective, significantly improving user satisfaction, presence, and ease of interaction while maintaining high immersion.

ABSTRACT

Head Mounted Displays (HMDs) allow users to experience virtual reality with a great level of immersion. However, even simple physical tasks like drinking a beverage can be difficult and awkward while in a virtual reality experience. We explore mixed reality renderings that selectively incorporate the physical world into the virtual world for interactions with physical objects. We conducted a user study comparing four rendering techniques that balances immersion in a virtual world with ease of interaction with the physical world. Finally, we discuss the pros and cons of each approach, suggesting guidelines for future rendering techniques that bring physical objects into virtual reality.

Motivation & Objective

• To address the challenge of performing simple physical tasks—like drinking—while wearing an immersive HMD, which blocks real-world awareness.
• To explore a design space of mixed reality renderings that selectively incorporate physical world elements into virtual environments to reduce context switching and immersion breaks.
• To evaluate how different visual fidelity and layout strategies for rendering physical elements affect user experience across diverse VR genres (e.g., movie, FPS, racing).
• To identify optimal rendering techniques that balance physical world awareness with sustained immersion in VR.

Proposed method

• Deployed a dual-webcam system to capture real-time stereo video of the physical environment, including the user’s hands and nearby objects.
• Implemented four distinct rendering techniques: (1) Object & Hands (OH), (2) Object, Hands & Edges (OHC), (3) Real World Windowed (RWW), and (4) Physical Picture-in-Picture (PPIP).
• Integrated the captured physical world data into the VR scene using one-to-one spatial scaling to preserve natural perception and proprioception.
• Conducted a user study with 18 participants across three VR experiences (movie, first-person shooter, racing game) to compare the four rendering techniques.
• Collected quantitative ratings (on 7-point scales) for overall satisfaction, immersion, presence, distraction, and ease of play, along with qualitative feedback.
• Analyzed results using statistical comparisons across conditions and VR genres to identify performance trends and user preferences.

Experimental results

Research questions

• RQ1 Which rendering technique best supports peripheral real-world interactions (e.g., retrieving a drink) while maintaining immersion in VR?
• RQ2 How do different levels of visual fidelity and spatial layout (e.g., windowed view vs. picture-in-picture) affect user perception of presence and distraction?
• RQ3 How does the effectiveness of each rendering technique vary across different VR experience types (e.g., low-engagement movie vs. high-engagement FPS)?
• RQ4 What role does contextual visual information—such as edges of surrounding surfaces—play in reducing user disorientation and improving interaction accuracy?
• RQ5 How do users perceive the trade-off between physical world awareness and immersion when using HMDs?

Key findings

• The Object & Hands with Context (OHC) rendering was consistently ranked highest in overall satisfaction, immersion, presence, and ease of play across all VR experiences.
• The OHC rendering, which displays hands, target objects, and salient edges of the environment, enabled users to locate and interact with physical objects (e.g., a cup) more quickly and confidently than other methods.
• The Object & Hands (OH) rendering, though minimal, was ranked second overall but showed high user variance, with some users reporting disorientation and difficulty locating objects.
• The Real World Windowed (RWW) and Physical Picture-in-Picture (PPIP) renderings were perceived as more immersion-breaking, with users describing a sense of 'limbo' and reduced connection to the virtual world.
• Lifting the HMD was the least preferred method, with users explicitly stating it broke immersion and was disruptive to the VR experience.
• Users preferred renderings that maintained one-to-one spatial scaling and ego-centric viewpoints, as these preserved natural perception and reduced cognitive load during physical interactions.