[Paper Review] ScreenAvoider: Protecting Computer Screens from Ubiquitous Cameras

Mohammed Korayem, Robert Templeman+3 more|arXiv (Cornell University)|Nov 28, 2014

Video Surveillance and Tracking Methods22 references13 citations

TL;DR

ScreenAvoider is a computer vision framework that detects and classifies images containing computer screens in lifelogging data, enabling users to enforce privacy policies based on screen presence and content. It uses deep learning for accurate detection and introduces ScreenTag to flag sensitive screen content like emails or messaging apps, significantly improving privacy management in ubiquitous camera environments.

ABSTRACT

We live and work in environments that are inundated with cameras embedded in devices such as phones, tablets, laptops, and monitors. Newer wearable devices like Google Glass, Narrative Clip, and Autographer offer the ability to quietly log our lives with cameras from a `first person' perspective. While capturing several meaningful and interesting moments, a significant number of images captured by these wearable cameras can contain computer screens. Given the potentially sensitive information that is visible on our displays, there is a need to guard computer screens from undesired photography. People need protection against photography of their screens, whether by other people's cameras or their own cameras. We present ScreenAvoider, a framework that controls the collection and disclosure of images with computer screens and their sensitive content. ScreenAvoider can detect images with computer screens with high accuracy and can even go so far as to discriminate amongst screen content. We also introduce a ScreenTag system that aids in the identification of screen content, flagging images with highly sensitive content such as messaging applications or email webpages. We evaluate our concept on realistic lifelogging datasets, showing that ScreenAvoider provides a practical and useful solution that can help users manage their privacy.

Motivation & Objective

• Address the growing privacy risk posed by wearable and ubiquitous cameras capturing sensitive information displayed on computer screens.
• Develop an automated system to detect when images contain computer screens, reducing reliance on manual review of large lifelog datasets.
• Enable users to define privacy policies based on screen presence and content type, especially for highly sensitive applications like email or messaging.
• Improve privacy management in lifelogging by distinguishing between images with screens and those with sensitive screen content.
• Provide a practical, deployable solution that integrates with existing cloud-based lifelogging services or operating systems for real-time policy enforcement.

Proposed method

• Utilizes deep convolutional neural networks (CNNs) to detect computer screens in images with high accuracy, leveraging state-of-the-art object detection techniques.
• Employs a two-stage detection pipeline: first identifying screen regions, then classifying the content within those regions using fine-grained visual recognition.
• Introduces ScreenTag, a component that identifies and flags images containing highly sensitive content such as email interfaces or messaging apps based on visual patterns.
• Applies attribute-based access control (ABAC) principles, where visual attributes (e.g., screen presence, content type) trigger privacy policies.
• Evaluates the system on realistic lifelogging datasets to simulate real-world deployment conditions and assess performance under practical constraints.
• Integrates with existing lifelogging platforms via APIs or OS-level services, enabling real-time policy enforcement during image collection or upload.

Experimental results

Research questions

• RQ1 Can a deep learning-based system accurately detect the presence of computer screens in images captured by wearable cameras?
• RQ2 To what extent can the system classify screen content into sensitive vs. non-sensitive categories, such as email, messaging, or general web browsing?
• RQ3 How effective is the ScreenTag system in identifying and flagging images with highly sensitive content like personal messages or financial data?
• RQ4 Can the framework be integrated into existing lifelogging pipelines to enforce privacy policies without significant user intervention?
• RQ5 How does the system perform in real-world lifelogging scenarios involving large volumes of images with mixed content?

Key findings

• ScreenAvoider achieves high accuracy in detecting computer screens in lifelogging images, demonstrating the feasibility of automated screen detection in real-world settings.
• The system successfully distinguishes between images containing screens and those without, enabling effective policy enforcement based on screen presence.
• ScreenTag effectively identifies and flags images with sensitive content such as messaging applications and email webpages, reducing the risk of unintended disclosure.
• The framework enables users to enforce privacy policies that block or restrict cloud storage of images containing screens, especially those with sensitive content.
• Evaluation on realistic lifelogging datasets confirms that ScreenAvoider provides a practical and scalable solution for managing privacy in ubiquitous camera environments.
• The integration of deep learning with attribute-based access control allows for semantic-level privacy management, marking a significant advancement over previous rule-based or manual approaches.