[Paper Review] Robust Multi-Output Learning with Highly Incomplete Data via Restricted Boltzmann Machines

Giancarlo Fissore, Aurélien Decelle+2 more|arXiv (Cornell University)|Dec 19, 2019

Generative Adversarial Networks and Image Synthesis18 references5 citations

TL;DR

This paper proposes RBM-MO, a robust multi-output learning framework using Restricted Boltzmann Machines to jointly model incomplete features and partially observed labels via mean-field inference and contrastive divergence training. It achieves state-of-the-art performance in both transductive and inductive settings across multi-class and multi-label tasks, especially under high missing data rates, by leveraging complementary dependencies between features and labels.

ABSTRACT

In a standard multi-output classification scenario, both features and labels of training data are partially observed. This challenging issue is widely witnessed due to sensor or database failures, crowd-sourcing and noisy communication channels in industrial data analytic services. Classic methods for handling multi-output classification with incomplete supervision information usually decompose the problem into an imputation stage that reconstructs the missing training information, and a learning stage that builds a classifier based on the imputed training set. These methods fail to fully leverage the dependencies between features and labels. In order to take full advantage of these dependencies we consider a purely probabilistic setting in which the features imputation and multi-label classification problems are jointly solved. Indeed, we show that a simple Restricted Boltzmann Machine can be trained with an adapted algorithm based on mean-field equations to efficiently solve problems of inductive and transductive learning in which both features and labels are missing at random. The effectiveness of the approach is demonstrated empirically on various datasets, with particular focus on a real-world Internet-of-Things security dataset.

Motivation & Objective

• Address the challenge of multi-output classification when both features and labels are missing at random in real-world industrial data.
• Overcome limitations of existing methods that assume fully observed features during inference or are restricted to multi-label learning only.
• Develop a unified, probabilistic framework that jointly models features and labels to improve imputation and classification under high incompleteness.
• Enable effective inductive and transductive learning with incomplete training and test instances using a simple yet powerful generative model.
• Demonstrate robustness and generalization across diverse datasets, including a real-world IoT security dataset.

Proposed method

• Adapt the Restricted Boltzmann Machine (RBM) architecture to model joint distributions of features and labels under missing data.
• Use mean-field equations to approximate posterior distributions over hidden units during training and inference.
• Apply a contrastive divergence-based training procedure to efficiently optimize the model with incomplete observations.
• Implement a joint imputation and classification mechanism that leverages dependencies between missing features and labels.
• Support both transductive and inductive learning by applying the trained model to incomplete test instances without retraining.
• Ensure seamless applicability to both multi-class and multi-label classification through a unified probabilistic formulation.

Experimental results

Research questions

• RQ1 Can a simple generative model like the RBM effectively handle joint imputation of features and labels under high missingness?
• RQ2 How does joint modeling of features and labels improve classification and reconstruction accuracy compared to sequential imputation and learning?
• RQ3 To what extent does RBM-MO generalize across different data types, including multi-class and multi-label settings?
• RQ4 How does RBM-MO perform under extreme missingness rates (e.g., 80%) in both training and test data?
• RQ5 Can RBM-MO outperform state-of-the-art methods designed specifically for multi-label learning in multi-class scenarios with missing labels?

Key findings

• RBM-MO achieves significantly higher Hamming accuracy—up to 30% better—than baseline methods in transductive learning, especially at high missing rates.
• The model maintains low reconstruction error (RMSE) with standard deviation ≤0.01 across datasets, indicating high robustness and stability.
• On the MNIST dataset, RBM-MO shows superior reconstruction accuracy compared to baselines, particularly under high feature missingness.
• In inductive learning, RBM-MO consistently outperforms CLE and NoisyIMC, which fail on multi-class tasks due to structured missing label patterns.
• Even at 80% missing features and labels, RBM-MO produces the highest precision, enabling effective guidance for human analysts in real-world IoT threat detection.
• The method achieves state-of-the-art performance across public datasets and a real-world IoT security dataset, demonstrating strong generalization and practical usability.