[论文解读] On Embeddings for Numerical Features in Tabular Deep Learning

Yury Gorishniy, Ivan Rubachev+1 more|arXiv (Cornell University)|Mar 10, 2022

Model Reduction and Neural Networks被引用 58

一句话总结

本文提出将数值特征嵌入到表格DL中的方案（基于分段线性编码和周期性激活），并展示它们在多种骨干网络上显著提升性能，在若干基准数据集上缩小与GBDT的差距。

ABSTRACT

Recently, Transformer-like deep architectures have shown strong performance on tabular data problems. Unlike traditional models, e.g., MLP, these architectures map scalar values of numerical features to high-dimensional embeddings before mixing them in the main backbone. In this work, we argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with GBDT on some traditionally GBDT-friendly benchmarks. We start by describing two conceptually different approaches to building embedding modules: the first one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations. Then, we empirically demonstrate that these two approaches can lead to significant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations. Importantly, we also show that embedding numerical features is beneficial for many backbones, not only for Transformers. Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures. Overall, we highlight embeddings for numerical features as an important design aspect with good potential for further improvements in tabular DL.

研究动机与目标

• Highlight embeddings for numerical features as a design choice in tabular DL.
• Develop and compare two embedding schemes (piecewise linear encoding and periodic activations).
• Show that embeddings improve performance across backbones (MLP, ResNet, Transformer) and on GBDT-friendly benchmarks.
• Demonstrate that simple MLPs can rival attention-based models when paired with proper embeddings.

提出的方法

• Formalize embeddings for numerical features as independent feature embeddings z_i = f_i(x_i^(num)).
• Propose two embedding blocks: (a) piecewise linear encoding (PLE) based on feature binning, (b) periodic activations with trainable pre-activation coefficients.
• Explain bin construction methods: quantile-based and target-aware (C4.5-inspired).
• Provide optional differentiable layers on top of embeddings (e.g., linear layers, ReLU).
• Adapt embeddings for both MLP-like and Transformer backbones; embeddings are concatenated or passed unchanged depending on backbone.
• Experiment with multiple backbones (MLP, ResNet, Transformer) and a broad set of embeddings to evaluate accuracy/RMSE across 11 datasets.

实验结果

研究问题

• RQ1 Do expressive embeddings for numerical features improve tabular DL performance beyond conventional linear blocks?
• RQ2 Are the benefits of numerical feature embeddings limited to Transformer-like models or universal across backbones?
• RQ3 Which embedding schemes (PLE quantile, PLE target-aware, periodic, or their combinations) yield the best performance across datasets?
• RQ4 Can simple MLPs with proper embeddings compete with state-of-the-art attention-based models and GBDT baselines?

主要发现

• Embedding numerical features yields notable gains for MLP, ResNet, and Transformer backbones.
• Piecewise linear encoding (PLE) often improves results, with quantile- and target-aware binning delivering strong performance.
• Periodic activations are effective and, with additional differentiable layers, often surpass plain periodic embeddings.
• MLP with proper embeddings can match or exceed Transformer-based models on several datasets, closing parts of the DL-GBDT gap.
• On several public benchmarks, DL models with embeddings achieve state-of-the-art among tabular DL methods and approach GBDT performance.
• Embedding overhead increases parameter count but can be offset by faster training with certain backbones.