[Paper Review] Finding Outliers in Surface Data and Video

Mia Hubert, Jakob Raymaekers+2 more|arXiv (Cornell University)|Jan 29, 2016

Spectroscopy and Chemometric Analyses13 references3 citations

TL;DR

This paper proposes a robust method for detecting outliers in surface, image, and video data using functional adjusted outlyingness (fAO) on raw or residual data after multiway modeling. By computing outlyingness across grid points and visualizing results via functional outlier maps (FOMs) and heatmaps, the method effectively identifies deviant regions in fluorescence spectra, MRI scans, and surveillance videos, with clear detection of transient objects like a person moving through a scene.

ABSTRACT

Surface, image and video data can be considered as functional data with a bivariate domain. To detect outlying surfaces or images, a new method is proposed based on the mean and the variability of the degree of outlyingness at each grid point. A rule is constructed to flag the outliers in the resulting functional outlier map. Heatmaps of their outlyingness indicate the regions which are most deviating from the regular surfaces. The method is applied to fluorescence excitation-emission spectra after fitting a PARAFAC model, to MRI image data which are augmented with their gradients, and to video surveillance data.

Motivation & Objective

• To develop a robust, distribution-free method for detecting outliers in functional data with a bivariate domain, such as surfaces, images, and videos.
• To improve outlier detection by applying the functional adjusted outlyingness (fAO) to residuals from a PARAFAC model rather than raw data, enhancing sensitivity to subtle deviations.
• To extend the functional outlier map (FOM) and outlyingness heatmap visualization to multivariate functional data with bivariate domains for interpretable outlier localization.
• To demonstrate the method’s effectiveness on real-world datasets including excitation-emission fluorescence matrices, MRI images with gradients, and surveillance video sequences.

Proposed method

• The method uses functional adjusted outlyingness (fAO) defined as a weighted sum of univariate adjusted outlyingness (AO) values across all grid points in a surface or image, with uniform weights by default.
• The AO at each grid point is computed as a robust measure of deviation from the median, accounting for skewness by estimating scale separately on each side of the median.
• For multiway data, a PARAFAC model is fitted to the data, and fAO is computed on the residuals to improve detection of structural outliers.
• The method incorporates image gradients to enhance detection of edges and boundaries, which are often sources of deviation in medical and surveillance images.
• A cutoff rule is introduced to distinguish between regular and outlying surfaces based on the fAO value, enabling objective flagging of outliers.
• Visualization is achieved through functional outlier maps (FOMs) and AO heatmaps, which show the degree and spatial distribution of outlyingness across the surface.

Experimental results

Research questions

• RQ1 How can functional adjusted outlyingness (fAO) be effectively extended to multivariate functional data with a bivariate domain such as images and videos?
• RQ2 To what extent does modeling the data with a PARAFAC model and analyzing residuals improve outlier detection compared to analyzing raw data?
• RQ3 Can the functional outlier map (FOM) and AO heatmaps successfully identify and localize deviant regions in complex real-world data such as MRI scans and surveillance videos?
• RQ4 How do the inclusion of image gradients and robust scale estimation enhance the detection of structural outliers in image data?

Key findings

• In the fluorescence excitation-emission dataset, the method successfully identified a single outlying spectrum after PARAFAC modeling, which was confirmed as an outlier due to its high fAO and vAO values.
• For MRI data, the AO heatmap revealed that the most outlying image (person 92) had a distinct region of high outlyingness near the brain's boundary, corresponding to a structural anomaly.
• The method detected the man entering and exiting the video frame by tracking changes in fAO and vAO values, with frames 484–487 showing the highest outlyingness due to a large fraction of outlying pixels.
• The AO heatmaps clearly localized the man’s position in each frame, with the darkest colors corresponding to pixels where the man’s body overlapped with the background, confirming spatial accuracy.
• The FOM successfully visualized the man’s path over time, with isolated outliers (frames 483–484) appearing just before and after his appearance, and sustained outliers (frames 488–500) indicating his presence in the scene.