On Embeddings for Numerical Features in Tabular Deep Learning

03/10/2022
by   Yura Gorishniy, et al.
0

Recently, Transformer-like deep architectures have shown strong performance on tabular data problems. Unlike traditional models, e.g., MLP, these architectures map scalar values of numerical features to high-dimensional embeddings before mixing them in the main backbone. In this work, we argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with GBDT on some traditionally GBDT-friendly benchmarks. We start by describing two conceptually different approaches to building embedding modules: the first one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations. Then, we empirically demonstrate that these two approaches can lead to significant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations. Importantly, we also show that embedding numerical features is beneficial for many backbones, not only for Transformers. Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures. Overall, we highlight that embeddings for numerical features are an important design aspect, which has good potential for further improvements in tabular DL.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/22/2021

Revisiting Deep Learning Models for Tabular Data

The necessity of deep learning for tabular data is still an unanswered q...
research
07/26/2023

TabR: Unlocking the Power of Retrieval-Augmented Tabular Deep Learning

Deep learning (DL) models for tabular data problems are receiving increa...
research
04/08/2021

Embeddings and Attention in Predictive Modeling

We explore in depth how categorical data can be processed with embedding...
research
07/07/2022

Revisiting Pretraining Objectives for Tabular Deep Learning

Recent deep learning models for tabular data currently compete with the ...
research
05/22/2018

A Tropical Approach to Neural Networks with Piecewise Linear Activations

We present a new, unifying approach following some recent developments o...
research
06/20/2023

Exploring the Performance and Efficiency of Transformer Models for NLP on Mobile Devices

Deep learning (DL) is characterised by its dynamic nature, with new deep...
research
04/16/2021

An Empirical Study of Extrapolation in Text Generation with Scalar Control

We conduct an empirical evaluation of extrapolation performance when con...

Please sign up or login with your details

Forgot password? Click here to reset