[Paper Review] All Reality: Virtual, Augmented, Mixed (X), Mediated (X,Y), and Multimediated Reality

Steve Mann, Thomas A. Furness+3 more|arXiv (Cornell University)|Apr 20, 2018

Virtual Reality Applications and Impacts23 references126 citations

TL;DR

The paper proposes a comprehensive taxonomy and continuum for realities (VR/AR/MR/XR/mediated/phenomenological) and introduces Multimediated Reality as a multidimensional, multisensory framework including extra senses and social/surveillance considerations.

ABSTRACT

The contributions of this paper are: (1) a taxonomy of the "Realities" (Virtual, Augmented, Mixed, Mediated, etc.), and (2) some new kinds of "reality" that come from nature itself, i.e. that expand our notion beyond synthetic realities to include also phenomenological realities. VR (Virtual Reality) replaces the real world with a simulated experience (virtual world). AR (Augmented Reality) allows a virtual world to be experienced while also experiencing the real world at the same time. Mixed Reality provides blends that interpolate between real and virtual worlds in various proportions, along a "Virtuality" axis, and extrapolate to an "X-axis". Mediated Reality goes a step further by mixing/blending and also modifying reality. This modifying of reality introduces a second axis. Mediated Reality is useful as a seeing aid (e.g. modifying reality to make it easier to understand), and for psychology experiments like Stratton's 1896 upside-down eyeglasses experiment. We propose Multimediated Reality as a multidimensional multisensory mediated reality that includes not just interactive multimedia-based reality for our five senses, but also includes additional senses (like sensory sonar, sensory radar, etc.), as well as our human actions/actuators. These extra senses are mapped to our human senses using synthetic synesthesia. This allows us to directly experience real (but otherwise invisible) phenomena, such as wave propagation and wave interference patterns, so that we can see radio waves and sound waves and how they interact with objects and each other. Multimediated reality is multidimensional, multimodal, multisensory, and multiscale. It is also multidisciplinary, in that we must consider not just the user, but also how the technology affects others, e.g. how its physical appearance affects social situations.

Motivation & Objective

• Define a taxonomy and framework for Realities including VR, AR, MR, XR, mediated reality, and multimediated reality.
• Extend traditional reality concepts to include phenomenological and multisensory experiences beyond synthetic senses.
• Introduce the idea of Multimediated Reality as a multidimensional system that maps extra senses and human actions to human perception.
• Highlight social, ethical, and infrastructural considerations of multimediated reality (e.g., privacy and veillance).

Proposed method

• Review and synthesize historical and contemporary concepts of VR/AR/MR/XR and mediated reality.
• Propose a multidimensional continuum (Reality–Virtuality, Mediality, Phenomenality) to classify emerging realities.
• Introduce hardware/software constructs (e.g., phenomology-based visualization, SWIM-like sensing) to enable direct perception of physical phenomena (waves, interference).
• Describe Multimediated Reality as an integration of multisensory synesthesia and human-in-the-loop intelligence (HI) in a multiveillant and multiscale ecosystem.
• Present a continuum model with multiple axes (Reality–Virtuality, Mediality, Phenomenality, Veillance, Wearability, etc.) and discuss implications for design and ethics.

Experimental results

Research questions

• RQ1 What taxonomy and continuum best capture VR, AR, MR, XR, mediated reality, and phenomenological realities?
• RQ2 How can multimediated reality be realized as a multidimensional, multisensory framework that includes extra senses and human actions?
• RQ3 What are the social, privacy, and ethical implications of deploying multimediated reality across wearables and smart environments?
• RQ4 How can mediated/phenomenological approaches complement traditional measurement and education tools?

Key findings

• A formal taxonomy and a multidimensional continuum are proposed to unify Virtual Reality, Augmented Reality, Mixed Reality, XR, and Mediated Reality.
• Multimediated Reality extends beyond synthetic senses to include extra senses (e.g., radio, sonar) via synthetic synesthesia, enabling direct visualization of physical phenomena.
• The framework emphasizes multiscale, multisensory experiences and incorporates social and veillance considerations (surveillance and sousveillance).
• Phenomenological Reality is introduced as a way to visualize real physical quantities directly through nature-based signals (e.g., wave propagation).
• HI (Humanistic Intelligence) and a feedback-loop approach are proposed as core to intelligent, responsible multimediated systems.
• The continuum integrates sensory attenuation, environmental context (smart buildings/cities), and ethical dimensions for future design.