[论文解读] All Reality: Virtual, Augmented, Mixed (X), Mediated (X,Y), and Multimediated Reality

Steve Mann, Thomas A. Furness+3 more|arXiv (Cornell University)|Apr 20, 2018

Virtual Reality Applications and Impacts参考文献 23被引用 126

一句话总结

本文提出一个关于现实的全面分类法与连续体（VR/AR/MR/XR/mediated/phenomenological）并引入 Multimediated Reality，作为一个多维度、多感官的框架，包含额外感官和社会/监控方面的考量。

ABSTRACT

The contributions of this paper are: (1) a taxonomy of the "Realities" (Virtual, Augmented, Mixed, Mediated, etc.), and (2) some new kinds of "reality" that come from nature itself, i.e. that expand our notion beyond synthetic realities to include also phenomenological realities. VR (Virtual Reality) replaces the real world with a simulated experience (virtual world). AR (Augmented Reality) allows a virtual world to be experienced while also experiencing the real world at the same time. Mixed Reality provides blends that interpolate between real and virtual worlds in various proportions, along a "Virtuality" axis, and extrapolate to an "X-axis". Mediated Reality goes a step further by mixing/blending and also modifying reality. This modifying of reality introduces a second axis. Mediated Reality is useful as a seeing aid (e.g. modifying reality to make it easier to understand), and for psychology experiments like Stratton's 1896 upside-down eyeglasses experiment. We propose Multimediated Reality as a multidimensional multisensory mediated reality that includes not just interactive multimedia-based reality for our five senses, but also includes additional senses (like sensory sonar, sensory radar, etc.), as well as our human actions/actuators. These extra senses are mapped to our human senses using synthetic synesthesia. This allows us to directly experience real (but otherwise invisible) phenomena, such as wave propagation and wave interference patterns, so that we can see radio waves and sound waves and how they interact with objects and each other. Multimediated reality is multidimensional, multimodal, multisensory, and multiscale. It is also multidisciplinary, in that we must consider not just the user, but also how the technology affects others, e.g. how its physical appearance affects social situations.

研究动机与目标

• 为现实（包括 VR、AR、MR、XR、mediated reality 和 multimediated reality）定义一个分类法和框架。
• 将传统现实概念扩展到包含现象学体验和超越合成感官的多感官体验。
• 提出 Multimediated Reality 的概念，作为一个多维度系统，将额外感官和人类行为映射到人类感知。
• 强调 Multimediated Reality 的社会、伦理和基础设施考量（如隐私与监视/监控 veillance）。

提出的方法

• 回顾并综合历史与当代关于 VR/AR/MR/XR 及 mediated reality 的概念。
• 提出一个多维连续体（Reality–Virtuality、Mediality、Phenomenality）来对新兴现实进行分类。
• 引入硬件/软件结构（如基于现象学的可视化、SWIM 风格感知）以实现对物理现象（波、干涉）等的直接感知。
• 将 Multimediated Reality 描述为在多感官联觉与人类在环智能（HI）在多层级和多尺度生态系统中的整合。
• 提出一个具有多条轴线的连续体模型（Reality–Virtuality、Mediality、Phenomenality、Veillance、Wearability 等），并讨论对设计与伦理的影响。

实验结果

研究问题

• RQ1 什么样的分类法和连续体能够最好地捕捉 VR、AR、MR、XR、mediated reality 和 phenomenological realities？
• RQ2 如何将 multimediated reality 实现为一个包含额外感官和人类行为的多维度、多感官框架？
• RQ3 在可穿戴设备和智能环境中部署 multimediated reality 的社会、隐私与伦理含义是什么？
• RQ4 mediated/phenomenological 方法如何补充传统测量和教育工具？

主要发现

• 提出了一个正式的分类法和一个多维连续体，以统一 Virtual Reality、Augmented Reality、Mixed Reality、XR、以及 Mediated Reality。
• Multimediated Reality 超越合成感官，通过合成共感（synthetic synesthesia）包含额外感官（如无线电、声纳），实现对物理现象的直接可视化。
• 该框架强调多尺度、多感官体验，并融入社会与 veillance 考量（监视与 sousveillance）。
• Phenomenological Reality 被引入作为通过自然信号（如波传播）直观可视化真实物理量的一种方式。
• 提出 HI（Humanistic Intelligence）及反馈循环方法作为智能、负责任的多模态系统的核心。
• 该连续体整合感官衰减、环境上下文（智能建筑/城市）及未来设计的伦理维度。