[Paper Review] Unconventional TV Detection using Mobile Devices

Mohamed Ibrahim, Ahmed Saeed+2 more|arXiv (Cornell University)|May 17, 2013

Music and Audio Processing12 references3 citations

TL;DR

This paper proposes a mobile-device-based system that detects TV presence, power state, and active channel using acoustic and visual sensing from smartphones. By fusing microphone and camera data with sensor fusion logic, it achieves a 0.978 F-measure in controlled environments, enabling fine-grained TV viewing context detection without specialized hardware.

ABSTRACT

Recent studies show that the TV viewing experience is changing giving the rise of trends like "multi-screen viewing" and "connected viewers". These trends describe TV viewers that use mobile devices (e.g. tablets and smart phones) while watching TV. In this paper, we exploit the context information available from the ubiquitous mobile devices to detect the presence of TVs and track the media being viewed. Our approach leverages the array of sensors available in modern mobile devices, e.g. cameras and microphones, to detect the location of TV sets, their state (ON or OFF), and the channels they are currently tuned to. We present the feasibility of the proposed sensing technique using our implementation on Android phones with different realistic scenarios. Our results show that in a controlled environment a detection accuracy of 0.978 F-measure could be achieved.

Motivation & Objective

• Address the lack of scalable, fine-grained TV viewing context detection in the era of multi-screen and connected viewers.
• Overcome limitations of traditional TV detection methods relying on set-top boxes or power leakage sensors, which are not scalable.
• Leverage ubiquitous mobile sensors (camera, microphone) to detect TV presence, state (ON/OFF), and current channel without requiring dedicated hardware.
• Differentiate TV audio and video signals from similar sources like laptops, people talking, or picture frames to reduce false positives.
• Enable privacy-preserving, energy-efficient TV context detection through local processing and adaptive sensing rates.

Proposed method

• Collect audio and video sensory data from Android smartphones using built-in microphones and cameras.
• Extract acoustic fingerprints using spectral features from 30-second audio clips, trained on 91 recordings and tested on 60 independent samples.
• Detect dynamic visual content on screens using background subtraction and contour analysis, filtering for rectangle-shaped regions via contour simplification and convex hull detection.
• Apply a Ramer-Douglas-Peucker algorithm to reduce contour points and identify four-point convex hulls as candidate rectangles.
• Fuse detection results from audio and visual modules using an OR rule to prioritize minimizing false negatives, with decision control based on sensor sampling frequency.
• Implement a Detection Decision Controller to manage sensing frequency and energy efficiency, and support offloading computation to nearby devices for battery conservation.

Experimental results

Research questions

• RQ1 Can mobile device sensors (microphone and camera) reliably detect the presence of a TV set in real-world environments?
• RQ2 How accurately can acoustic and visual features distinguish TV signals from similar sources like human speech or laptop displays?
• RQ3 What is the optimal fusion strategy between audio and visual detection to minimize false negatives while controlling false positives?
• RQ4 To what extent can the system maintain high accuracy while ensuring energy efficiency and user privacy?
• RQ5 How do variations in audio sampling rate and frame capture frequency affect detection performance and system efficiency?

Key findings

• The acoustic detection method achieved a 0.928 F-measure with a 0.13 false negative rate and zero false positive rate, primarily failing when TV talk shows were confused with human conversation.
• The visual detection method achieved a 0.933 F-measure with zero false negatives and a 0.125 false positive rate, mainly due to misclassification of non-TV rectangular objects.
• The fused detection approach achieved a 0.978 F-measure with zero false negatives and a 0.042 false positive rate, demonstrating superior performance through sensor fusion.
• The system's false negative rate was minimized by using an OR-based fusion rule, ensuring that a TV is detected even if one sensor fails.
• The detection accuracy was sensitive to sensor configuration: increasing audio sampling rate reduced false negatives, and increasing frame count improved visual detection reliability.
• The system supports privacy-preserving deployment by enabling local processing and anonymized data forwarding, reducing exposure of raw sensory data.