[Paper Review] Multimodal vs. Unimodal Physiological Control in Videogames for Enhanced Realism and Depth

Gonçalo Filipe Lopes Coelho Amaral da Silva|arXiv (Cornell University)|Jun 2, 2014

Virtual Reality Applications and Impacts44 references3 citations

TL;DR

This paper proposes multimodal direct biofeedback in first-person shooters, combining two physiological sensors (e.g., respiration and temperature) per game mechanic to enhance realism and depth. Results show that while unimodal biofeedback improves fun and realism over standard controls, multimodal biofeedback significantly increases perceived depth, realism, and player empowerment—though at the cost of higher physical and cognitive load.

ABSTRACT

(arXiv abridged abstract) In the last two decades, videogames have evolved in a nearly explosive way from the pixelated graphics to today's near-realistic 3D environments. The interaction devices traditionally used in videogames have not evolved with the same intensity, but recent HCI studies have explored biofeedback interaction - the explicit manipulation of a person's physiological data as input to a system - as an alternative to them. Traditional biofeedback prototypes apply 1 sensor to each game mechanic (unimodality). In this dissertation, we introduce the combination of 2 physiological sensors simultaneously per game mechanic (multimodality) and present a First-Person Shooter game comprised of 8 game mechanics with three interaction flavours (no biofeedback/vanilla, unimodal and multimodal). An empirical study with 32 regular players was employed to explore and study differences between the three interaction types and where they can be best employed. Players compared the three games in terms of Fun, Ease of Use, Originality, Playability and Favourite Condition. For the sake of completeness, other evaluation methods were used as well: IMI Questionnaire, keywords association and open-ended commentaries. The vanilla version was considered easier to use, but both biofeedback versions were considered the most fun. Both versions were praised differently: the unimodal version for its simplicity of use, and the multimodal for its realism, activation safety of game mechanics and depth added to the game. Our conclusion is that multimodal biofeedback can have a relevant impact in terms of added depth, depending on the way it is used inside the game. On a boundary case, it can be used to increase the feeling of empowerment on the player when using certain abilities, or to intentionally make in-game actions more difficult by demanding more physical effort from the player.

Motivation & Objective

• To address the limitations of unimodal biofeedback, which oversimplifies player interaction and fails to justify replacing traditional input methods.
• To investigate whether combining multiple physiological sensors per game mechanic (multimodal biofeedback) enhances realism, depth, and player engagement.
• To compare multimodal biofeedback with standard keyboard/mouse and unimodal biofeedback in terms of fun, usability, originality, and playability.
• To evaluate how different biofeedback types affect player perception, especially regarding immersion, realism, and physical effort.
• To explore the design implications of context-sensitive biofeedback mechanics and long-term usability challenges.

Proposed method

• Developed a first-person shooter prototype using the Unreal Development Kit with three control variants: vanilla (keyboard/mouse), unimodal biofeedback (one sensor per mechanic), and multimodal biofeedback (two sensors per mechanic).
• Integrated physiological sensors measuring respiration (RESP) and temperature (TEMP) to control in-game actions such as recoil reduction and underwater exploration.
• Designed mechanics where multimodal control required coordinated physiological inputs (e.g., breath-holding + temperature change for stealth or submersion).
• Conducted user testing with 32 participants who compared all three versions across multiple metrics: fun, ease of use, originality, playability, and favorite condition.
• Collected qualitative feedback via keyword sorting and open-ended commentaries, and administered the IMI questionnaire to assess intrinsic motivation and perceived enjoyment.
• Analyzed results using comparative scoring and thematic analysis to assess differences between unimodal and multimodal biofeedback in perceived depth and realism.

Experimental results

Research questions

• RQ1 Does multimodal direct biofeedback significantly increase perceived realism and depth compared to unimodal or standard control schemes?
• RQ2 How do players rate the fun, usability, and playability of multimodal biofeedback versus unimodal and vanilla control schemes?
• RQ3 To what extent does combining multiple physiological signals per mechanic enhance immersion and player empowerment?
• RQ4 Are there perceptible differences in player confusion or activation difficulty between unimodal and multimodal biofeedback, especially when mechanics share sensor inputs?
• RQ5 What are the long-term usability and physical strain implications of sustained biofeedback gameplay?

Key findings

• Both unimodal and multimodal biofeedback versions were rated significantly more fun than the vanilla keyboard/mouse version, with no statistically significant difference in overall fun or playability between the two biofeedback types.
• Multimodal biofeedback was perceived as more realistic, deeper, and safer to activate than unimodal biofeedback, with players appreciating the added physical and cognitive engagement.
• Players reported that multimodal biofeedback enhanced their sense of empowerment and immersion, especially when requiring coordinated physiological actions (e.g., breath-holding and temperature change for stealth or submersion).
• The IMI questionnaire confirmed a significant positive impact of biofeedback on intrinsic motivation and enjoyment, likely due to the direct reflection of bodily actions in-game.
• Some participants reported confusion in unimodal versions when the same sensor (e.g., RESP) controlled multiple unrelated mechanics (e.g., breath-holding underwater and invisibility), which was resolved in the multimodal version through sensor separation.
• The study suggests that combining unimodal and multimodal biofeedback based on in-game context (e.g., light vs. heavy object handling) can optimize gameplay depth and player experience.