Scale-invariant representation of machine learning

09/07/2021
by   Sungyeop Lee, et al.
0

The success of machine learning stems from its structured data representation. Similar data have close representation as compressed codes for classification or emerged labels for clustering. We observe that the frequency of the internal representation follows power laws in both supervised and unsupervised learning. The scale-invariant distribution implies that machine learning largely compresses frequent typical data, and at the same time, differentiates many atypical data as outliers. In this study, we derive how the power laws can naturally arise in machine learning. In terms of information theory, the scale-invariant representation corresponds to a maximally uncertain data grouping among possible representations that guarantee pre-specified learning accuracy.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/11/2022

Investigating Power laws in Deep Representation Learning

Representation learning that leverages large-scale labelled datasets, is...
research
09/17/2020

The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons from Infant Learning

After a surge in popularity of supervised Deep Learning, the desire to r...
research
04/05/2019

Is 'Unsupervised Learning' a Misconceived Term?

Is all of machine learning supervised to some degree? The field of machi...
research
04/20/2021

Permutation-Invariant Variational Autoencoder for Graph-Level Representation Learning

Recently, there has been great success in applying deep neural networks ...
research
05/15/2019

Embeddings and Representation Learning for Structured Data

Performing machine learning on structured data is complicated by the fac...
research
07/24/2019

Learning abstract perceptual notions: the example of space

Humans are extremely swift learners. We are able to grasp highly abstrac...
research
12/19/2017

Scale-invariant temporal history (SITH): optimal slicing of the past in an uncertain world

In both the human brain and any general artificial intelligence (AI), a ...

Please sign up or login with your details

Forgot password? Click here to reset