DeepAI AI Chat
Log In Sign Up

Maximum likelihood method revisited: Gauge symmetry in Kullback – Leibler divergence and performance-guaranteed regularization

by   Akihisa Ichiki, et al.

The maximum likelihood method is the best-known method for estimating the probabilities behind the data. However, the conventional method obtains the probability model closest to the empirical distribution, resulting in overfitting. Then regularization methods prevent the model from being excessively close to the wrong probability, but little is known systematically about their performance. The idea of regularization is similar to error-correcting codes, which obtain optimal decoding by mixing suboptimal solutions with an incorrectly received code. The optimal decoding in error-correcting codes is achieved based on gauge symmetry. We propose a theoretically guaranteed regularization in the maximum likelihood method by focusing on a gauge symmetry in Kullback – Leibler divergence. In our approach, we obtain the optimal model without the need to search for hyperparameters frequently appearing in regularization.


page 1

page 2

page 3

page 4


qecGPT: decoding Quantum Error-correcting Codes with Generative Pre-trained Transformers

We propose a general framework for decoding quantum error-correcting cod...

Distance Enumerators for Number-Theoretic Codes

The number-theoretic codes are a class of codes defined by single or mul...

Bounds on the Error Probability of Raptor Codes under Maximum Likelihood Decoding

In this paper upper bounds on the probability of decoding failure under ...

Analysis of overfitting in the regularized Cox model

The Cox proportional hazards model is ubiquitous in the analysis of time...

Optimal Upward Scaling of Minimum-TSC Binary Signature Sets

We consider upward scaling an overloaded min-TSC binary signature set an...

Complex Structure Leads to Overfitting: A Structure Regularization Decoding Method for Natural Language Processing

Recent systems on structured prediction focus on increasing the level of...