SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

06/02/2021
by   Gowthami Somepalli, et al.
1

Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-supervised pre-training method for use when labels are scarce. SAINT consistently improves performance over previous deep learning methods, and it even outperforms gradient boosting methods, including XGBoost, CatBoost, and LightGBM, on average over a variety of benchmark tasks.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 9

page 18

11/04/2021

Leveraging Time Irreversibility with Order-Contrastive Pre-training

Label-scarce, high-dimensional domains such as healthcare present a chal...
10/15/2021

Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science

Deep learning techniques have been increasingly applied to the natural s...
05/27/2020

CLOCS: Contrastive Learning of Cardiac Signals

The healthcare industry generates troves of unlabelled physiological dat...
11/14/2016

Post Training in Deep Learning with Last Kernel

One of the main challenges of deep learning methods is the choice of an ...
11/06/2021

Towards noise robust trigger-word detection with contrastive learning pre-task for fast on-boarding of new trigger-words

Trigger-word detection plays an important role as the entry point of use...
11/30/2015

Cost-aware Pre-training for Multiclass Cost-sensitive Deep Learning

Deep learning has been one of the most prominent machine learning techni...
07/10/2020

Contrastive Training for Improved Out-of-Distribution Detection

Reliable detection of out-of-distribution (OOD) inputs is increasingly u...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.