Stochastic gradient descent with gradient estimator for categorical features

09/08/2022
by   Paul Peseux, et al.
0

Categorical data are present in key areas such as health or supply chain, and this data require specific treatment. In order to apply recent machine learning models on such data, encoding is needed. In order to build interpretable models, one-hot encoding is still a very good solution, but such encoding creates sparse data. Gradient estimators are not suited for sparse data: the gradient is mainly considered as zero while it simply does not always exists, thus a novel gradient estimator is introduced. We show what this estimator minimizes in theory and show its efficiency on different datasets with multiple model architectures. This new estimator performs better than common estimators under similar settings. A real world retail dataset is also released after anonymization. Overall, the aim of this paper is to thoroughly consider categorical data and adapt models and optimizers to these key features.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/03/2016

Categorical Reparameterization with Gumbel-Softmax

Categorical variables are a natural choice for representing discrete str...
research
12/22/2021

Evaluating categorical encoding methods on a real credit card fraud detection database

Correctly dealing with categorical data in a supervised learning context...
research
12/13/2014

The Statistics of Streaming Sparse Regression

We present a sparse analogue to stochastic gradient descent that is guar...
research
08/02/2022

A Note on Zeroth-Order Optimization on the Simplex

We construct a zeroth-order gradient estimator for a smooth function def...
research
11/05/2019

A Rule for Gradient Estimator Selection, with an Application to Variational Inference

Stochastic gradient descent (SGD) is the workhorse of modern machine lea...
research
10/25/2022

Unsupervised Anomaly Detection for Auditing Data and Impact of Categorical Encodings

In this paper, we introduce the Vehicle Claims dataset, consisting of fr...
research
10/04/2022

Representing missing values through polar encoding

We propose polar encoding, a representation of categorical and numerical...

Please sign up or login with your details

Forgot password? Click here to reset