[Paper Review] On Embeddings for Numerical Features in Tabular Deep Learning

Yury Gorishniy, Ivan Rubachev+1 more|arXiv (Cornell University)|Mar 10, 2022

Model Reduction and Neural Networks58 citations

TL;DR

The paper introduces embedding schemes for numerical features in tabular DL (piecewise linear encoding and periodic activations) and shows they significantly boost performance across backbones, closing the gap with GBDT on several benchmarks.

ABSTRACT

Recently, Transformer-like deep architectures have shown strong performance on tabular data problems. Unlike traditional models, e.g., MLP, these architectures map scalar values of numerical features to high-dimensional embeddings before mixing them in the main backbone. In this work, we argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with GBDT on some traditionally GBDT-friendly benchmarks. We start by describing two conceptually different approaches to building embedding modules: the first one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations. Then, we empirically demonstrate that these two approaches can lead to significant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations. Importantly, we also show that embedding numerical features is beneficial for many backbones, not only for Transformers. Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures. Overall, we highlight embeddings for numerical features as an important design aspect with good potential for further improvements in tabular DL.

Motivation & Objective

• Highlight embeddings for numerical features as a design choice in tabular DL.
• Develop and compare two embedding schemes (piecewise linear encoding and periodic activations).
• Show that embeddings improve performance across backbones (MLP, ResNet, Transformer) and on GBDT-friendly benchmarks.
• Demonstrate that simple MLPs can rival attention-based models when paired with proper embeddings.

Proposed method

• Formalize embeddings for numerical features as independent feature embeddings z_i = f_i(x_i^(num)).
• Propose two embedding blocks: (a) piecewise linear encoding (PLE) based on feature binning, (b) periodic activations with trainable pre-activation coefficients.
• Explain bin construction methods: quantile-based and target-aware (C4.5-inspired).
• Provide optional differentiable layers on top of embeddings (e.g., linear layers, ReLU).
• Adapt embeddings for both MLP-like and Transformer backbones; embeddings are concatenated or passed unchanged depending on backbone.
• Experiment with multiple backbones (MLP, ResNet, Transformer) and a broad set of embeddings to evaluate accuracy/RMSE across 11 datasets.

Experimental results

Research questions

• RQ1 Do expressive embeddings for numerical features improve tabular DL performance beyond conventional linear blocks?
• RQ2 Are the benefits of numerical feature embeddings limited to Transformer-like models or universal across backbones?
• RQ3 Which embedding schemes (PLE quantile, PLE target-aware, periodic, or their combinations) yield the best performance across datasets?
• RQ4 Can simple MLPs with proper embeddings compete with state-of-the-art attention-based models and GBDT baselines?

Key findings

• Embedding numerical features yields notable gains for MLP, ResNet, and Transformer backbones.
• Piecewise linear encoding (PLE) often improves results, with quantile- and target-aware binning delivering strong performance.
• Periodic activations are effective and, with additional differentiable layers, often surpass plain periodic embeddings.
• MLP with proper embeddings can match or exceed Transformer-based models on several datasets, closing parts of the DL-GBDT gap.
• On several public benchmarks, DL models with embeddings achieve state-of-the-art among tabular DL methods and approach GBDT performance.
• Embedding overhead increases parameter count but can be offset by faster training with certain backbones.