[Paper Review] MET: Masked Encoding for Tabular Data

Kushal Majmundar, Sachin Goyal+2 more|arXiv (Cornell University)|Jun 17, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

MET proposes a novel, reconstruction-based self-supervised learning method for tabular data that eliminates the need for data augmentations by using masked encoding with coordinate-specific representations and adversarial reconstruction loss. It achieves state-of-the-art performance, improving downstream classification accuracy by up to 9% over prior methods on five diverse tabular datasets.

ABSTRACT

We consider the task of self-supervised representation learning (SSL) for tabular data: tabular-SSL. Typical contrastive learning based SSL methods require instance-wise data augmentations which are difficult to design for unstructured tabular data. Existing tabular-SSL methods design such augmentations in a relatively ad-hoc fashion and can fail to capture the underlying data manifold. Instead of augmentations based approaches for tabular-SSL, we propose a new reconstruction based method, called Masked Encoding for Tabular Data (MET), that does not require augmentations. MET is based on the popular MAE approach for vision-SSL [He et al., 2021] and uses two key ideas: (i) since each coordinate in a tabular dataset has a distinct meaning, we need to use separate representations for all coordinates, and (ii) using an adversarial reconstruction loss in addition to the standard one. Empirical results on five diverse tabular datasets show that MET achieves a new state of the art (SOTA) on all of these datasets and improves up to 9% over current SOTA methods. We shed more light on the working of MET via experiments on carefully designed simple datasets.

Motivation & Objective

• To address the challenge of designing effective data augmentations for unstructured tabular data in self-supervised learning.
• To develop a reconstruction-based alternative to contrastive learning for tabular-SSL that avoids reliance on ad-hoc or domain-specific augmentations.
• To improve representation learning in tabular data by leveraging coordinate-specific embeddings and adversarial training.
• To demonstrate that concatenating all token representations, rather than pooling, leads to better downstream performance in tabular SSL.
• To investigate whether reconstruction-based methods can learn linearly separable representations in non-linear data distributions.

Proposed method

• MET uses a transformer-based encoder-decoder architecture to reconstruct masked features in tabular data.
• It applies masking to a fraction of features and uses a learnable mask token for all masked coordinates, with positional encodings for each feature.
• Instead of average pooling, MET concatenates the representations of all features for downstream fine-tuning, preserving coordinate-specific information.
• An adversarial reconstruction loss is introduced by performing gradient ascent on the input space to generate perturbations that improve robustness.
• The model is trained end-to-end using a combination of standard masked reconstruction loss and adversarial reconstruction loss.
• The method avoids instance-level data augmentations, relying solely on reconstruction and adversarial regularization.

Experimental results

Research questions

• RQ1 Can a reconstruction-based approach outperform contrastive learning methods in tabular self-supervised representation learning?
• RQ2 Does concatenating all token representations instead of pooling them lead to better downstream performance in tabular SSL?
• RQ3 Can adversarial training via gradient ascent on the reconstruction loss improve model generalization in tabular data?
• RQ4 Why do reconstruction-based methods learn linearly separable representations even in non-linear data distributions?
• RQ5 How does MET compare to existing SOTA methods like DACL, SubTab, and VIME across diverse tabular benchmarks?

Key findings

• MET achieves a new state-of-the-art performance on five standard tabular datasets, improving average accuracy by 3.2% over previous SOTA methods.
• On the FMNIST and CovType datasets, MET achieves 90.94% and 74.12% downstream accuracy, respectively, outperforming baseline methods.
• MET trained with only 20% of the labeled data can match the performance of supervised learning using the full labeled set on some datasets.
• Using separate mask tokens per feature improves performance on heterogeneous datasets like CovType (75.40% vs 74.12% for shared token), though at higher memory cost.
• Passing the mask token through the encoder degrades performance and increases computation, confirming that direct input to the decoder is preferable.
• Experiments on a simple synthetic dataset show that MET representations make non-linearly separable data linearly separable, suggesting improved disentanglement and generalization.