[Paper Review] The Effect of Visual Noise on The Completion of Security Critical Tasks

Tyler Kaczmarek, Alfed Kobsa+2 more|arXiv (Cornell University)|May 21, 2014

User Authentication and Security Systems22 references3 citations

TL;DR

This study investigates how unexpected visual and auditory noise affects performance on security-critical tasks, using an automated, unattended experimental setup with 147 participants. Surprisingly, visual and auditory distractions reduced error rates in Bluetooth pairing tasks, challenging assumptions that distractions uniformly impair security task performance.

ABSTRACT

User errors while performing security-critical tasks can lead to undesirable or even disastrous consequences. One major factor influencing mistakes and failures is complexity of such tasks, which has been studied extensively in prior research. Another important issue which hardly received any attention is the impact of both accidental and intended distractions on users performing security-critical tasks. In particular, it is unclear whether, and to what extent, unexpected sensory cues (e.g., auditory or visual) can influence user behavior and/or trigger mistakes. Better understanding of the effects of intended distractions will help clarify their role in adversarial models. As part of the research effort described in this paper, we administered a range of naturally occurring -- yet unexpected -- sounds while study participants attempted to perform a security-critical task. We found that, although these auditory cues lowered participants' failure rates, they had no discernible effect on their task completion times. To this end, we overview some relevant literature that explains these somewhat counter-intuitive findings. Conducting a thorough and meaningful study on user errors requires a large number of participants, since errors are typically infrequent and should not be instigated more than once per subject. To reduce the effort of running numerous subjects, we developed a novel experimental setup that was fully automated and unattended. We discuss our experience with this setup and highlight the pros and cons of generalizing its usage.

Motivation & Objective

• To investigate the impact of unexpected sensory distractions—particularly visual and auditory noise—on user performance in security-critical tasks.
• To address the challenge of studying rare user errors in usability research, which typically require large subject pools due to low failure rates.
• To develop and evaluate an automated, unattended experimental setup using a video-projected experimenter to reduce logistical and financial burdens.
• To explore whether adversarial sensory stimuli can be leveraged to improve task accuracy, contrary to conventional assumptions.
• To establish a scalable, ethically sound experimental model for large-scale usability studies in security-critical contexts.

Proposed method

• Deployed a fully automated, unattended experimental setup where a life-size, video-projected experimenter delivered instructions to participants.
• Conducted Bluetooth device pairing tasks in a public-like setting, with participants comparing a 6-digit PIN displayed on a Smartboard and their smartphone.
• Introduced unexpected auditory stimuli (e.g., sudden sounds) during task execution to simulate real-world distractions.
• Used video recordings and automated logging to collect performance data without live experimenter involvement.
• Implemented Institutional Review Board (IRB)-approved protocols to ensure ethical compliance, including data anonymization and video erasure.
• Collected data from 147 participants across multiple sessions, with minimal identifying information retained.

Experimental results

Research questions

• RQ1 How do unexpected auditory and visual distractions affect the success rate of users performing security-critical tasks?
• RQ2 Can unintended or adversarial sensory stimuli improve, rather than impair, performance in security tasks?
• RQ3 To what extent can an unattended, automated experimental setup reliably measure rare user errors in usability studies?
• RQ4 What are the ethical and logistical challenges of deploying large-scale, unattended experiments in public settings?
• RQ5 How do collaborative or multi-subject behaviors in shared experimental spaces affect data validity and reliability?

Key findings

• Unexpected auditory stimuli significantly reduced the failure rate in Bluetooth pairing tasks, contrary to the initial hypothesis that distractions would impair performance.
• Participants exposed to auditory noise completed tasks at a higher success rate than those in quiet conditions, indicating a potential facilitatory effect of noise.
• Task completion times were not significantly affected by the presence of auditory noise, suggesting that noise did not slow down participants.
• The automated, unattended experimental setup successfully collected data from 147 participants with minimal human intervention, validating its scalability.
• Multiple participants simultaneously engaging in the experiment was a common issue, with some collaborating, which introduced potential confounds in data collection.
• The study demonstrated that video recordings were essential for integrity checks, but also posed privacy and ethical considerations requiring careful management.