[Paper Review] Processing of missing data by neural networks

Marek Śmieja, Łukasz Struski+3 more|arXiv (Cornell University)|May 18, 2018

Neural Networks and Applications34 references68 citations

TL;DR

The paper introduces a general, theoretically justified approach to processing missing data in neural networks by replacing the first hidden layer neuron's response with its expectation over a missing-data density, enabling training with incomplete data and outperforming some imputation-based methods.

ABSTRACT

We propose a general, theoretically justified mechanism for processing missing data by neural networks. Our idea is to replace typical neuron's response in the first hidden layer by its expected value. This approach can be applied for various types of networks at minimal cost in their modification. Moreover, in contrast to recent approaches, it does not require complete data for training. Experimental results performed on different types of architectures show that our method gives better results than typical imputation strategies and other methods dedicated for incomplete data.

Motivation & Objective

• Motivate and formalize the challenge of learning from incomplete data in neural networks.
• Propose a probabilistic representation of missing data via conditional densities restricted to the observed subspace.
• Generalize the first-layer neuron activations to operate on probability distributions rather than concrete inputs.
• Show that the proposed generalization preserves information and achieves universal approximation properties.
• Demonstrate practical effectiveness across autoencoders, MLPs, and RBF networks on real datasets.

Proposed method

• Represent each incomplete data point (x, J) by a conditional density F_S on the affine subspace S corresponding to observed coordinates.
• Model the missing data density F with a mixture of diagonal Gaussians (GMM) and apply a regularization to compute conditional densities F_S.
• Generalize the activation of first-layer neurons to compute the expected output n(F_S) instead of n(x) for complete data.
• Derive closed-form expressions for the generalized ReLU and RBF activations under a Gaussian mixture input (Theorem 3.1 and Theorem 3.2).
• Explain how to train end-to-end by jointly learning the GMM parameters with network weights, using incomplete data in training.
• Argue and sketch a universal approximation result showing that ReLU-based networks with the generalized activations retain identifiability of measures (Theorem 4.1).

Experimental results

Research questions

• RQ1 How can neural networks process missing data without imputing single values?
• RQ2 Can we represent missing data via probabilistic densities and propagate them through networks without major architectural changes?
• RQ3 Do generalized first-layer activations preserve the information needed for universal function approximation?
• RQ4 How does training with incomplete data using the proposed density-based approach compare to standard imputation and context-encoder methods?

Key findings

• The method enables training neural networks on datasets containing only incomplete samples, avoiding the need for complete data.
• Generalized ReLU and RBF activations under Gaussian mixture inputs can be computed in closed form, enabling practical implementation.
• Experimental results across autoencoders, MLPs, and RBF networks show favorable performance versus standard imputation and some baseline methods.
• In MNIST-based reconstruction, the proposed method yields sharper results and lower error than k-NN, mean imputation, dropout, and some context-encoder baselines.
• On ESR EEG data, the method achieves competitive accuracy, often outperforming traditional imputation techniques, and matching context-encoder performance with incomplete training data.
• Across several UCI datasets with internal missing attributes, the approach frequently outperforms traditional imputation and direct kernel methods.