[Paper Review] Neural Relation Graph: A Unified Framework for Identifying Label Noise and Outlier Data

Jang-Hyun Kim, Sangdoo Yun+1 more|arXiv (Cornell University)|Jan 29, 2023

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes Neural Relation Graph, a unified framework that detects label noise and outlier data by modeling relational structures in deep feature embeddings. By constructing a data relation graph using similarity and compatibility scores, it outperforms state-of-the-art methods on ImageNet, ESC-50, and SST2, achieving SOTA performance in both label error and OOD detection with scalable, model-agnostic algorithms and an interactive visualization tool.

ABSTRACT

Diagnosing and cleaning data is a crucial step for building robust machine learning systems. However, identifying problems within large-scale datasets with real-world distributions is challenging due to the presence of complex issues such as label errors, under-representation, and outliers. In this paper, we propose a unified approach for identifying the problematic data by utilizing a largely ignored source of information: a relational structure of data in the feature-embedded space. To this end, we present scalable and effective algorithms for detecting label errors and outlier data based on the relational graph structure of data. We further introduce a visualization tool that provides contextual information of a data point in the feature-embedded space, serving as an effective tool for interactively diagnosing data. We evaluate the label error and outlier/out-of-distribution (OOD) detection performances of our approach on the large-scale image, speech, and language domain tasks, including ImageNet, ESC-50, and SST2. Our approach achieves state-of-the-art detection performance on all tasks considered and demonstrates its effectiveness in debugging large-scale real-world datasets across various domains. We release codes at https://github.com/snu-mllab/Neural-Relation-Graph.

Motivation & Objective

• To address the challenge of identifying label noise and outlier data in large-scale, real-world datasets with complex, overlapping issues.
• To overcome limitations of unary scoring methods that fail to distinguish between label errors and data outliers.
• To develop a scalable, model- and domain-agnostic framework for joint detection of label errors and out-of-distribution (OOD) data.
• To provide an interpretable visualization tool that reveals contextual relational patterns in the feature space for interactive data diagnosis.

Proposed method

• Construct a relational graph where nodes represent data points and edges encode similarity and compatibility between feature embeddings.
• Define a data relation function using a similarity kernel (e.g., cosine or RBF) combined with a compatibility term to measure relational strength.
• Use a temperature-regularized graph aggregation to compute global relational statistics (mean and variance) for each data point.
• Detect label errors by identifying data points with high variance in relations to their assigned labels.
• Detect outliers by identifying data points with low relational consistency across the entire graph structure.
• Introduce a data relation map visualization tool that displays mean and variance of relations for each sample, enabling interactive diagnosis.

Experimental results

Research questions

• RQ1 Can relational structure in the feature embedding space effectively distinguish between label errors and outlier data, where unary metrics fail?
• RQ2 How does the proposed relational graph approach compare to existing unary scoring methods in detecting label noise and OOD samples?
• RQ3 To what extent does the choice of similarity kernel and compatibility term affect detection performance across diverse domains?
• RQ4 Can the method generalize across different model architectures and data modalities (image, speech, text)?
• RQ5 How effective is the proposed visualization tool in supporting human-in-the-loop data diagnosis?

Key findings

• The proposed method achieves state-of-the-art performance on label error detection, with an AP of 0.526 on ImageNet under 8% noise ratio, outperforming the best baseline by 5.2%.
• On the same ImageNet benchmark, the method achieves a TNR95 of 0.695, significantly outperforming the best baseline (0.668) and demonstrating improved robustness.
• The method achieves SOTA performance on OOD detection for both ESC-50 and SST2, with high AUC and F1 scores, confirming its generalization across modalities.
• Ablation studies show that the compatibility term in the kernel function is critical for performance, as removing it still yields strong results, highlighting the framework's robustness.
• The data relation map visualization effectively reveals distinct relational patterns—e.g., high variance for label errors and low consistency for outliers—enabling accurate, interactive diagnosis.
• The method is scalable and model-agnostic, requiring only feature embeddings and predictions, making it deployable in real-world ML pipelines without retraining or gradient computation.