The simplicity bubble effect as a zemblanitous phenomenon in learning systems

by   Felipe S. Abrahão, et al.
University of Campinas

The ubiquity of Big Data and machine learning in society evinces the need of further investigation of their fundamental limitations. In this paper, we extend the “too-much-information-tends-to-behave-like-very-little-information” phenomenon to formal knowledge about lawlike universes and arbitrary collections of computably generated datasets. This gives rise to the simplicity bubble problem, which refers to a learning algorithm equipped with a formal theory that can be deceived by a dataset to find a locally optimal model which it deems to be the global one. However, the actual high-complexity globally optimal model unpredictably diverges from the found low-complexity local optimum. Zemblanity is defined by an undesirable but expected finding that reveals an underlying problem or negative consequence in a given model or theory, which is in principle predictable in case the formal theory contains sufficient information. Therefore, we argue that there is a ceiling above which formal knowledge cannot further decrease the probability of zemblanitous findings, should the randomly generated data made available to the learning algorithm and formal theory be sufficiently large in comparison to their joint complexity.


page 1

page 2

page 3

page 4


Algorithmic Probability of Large Datasets and the Simplicity Bubble Problem in Machine Learning

When mining large datasets in order to predict new data, limitations of ...

Grounding Occam's Razor in a Formal Theory of Simplicity

It is proposed that the Occam's Razor heuristic – when in doubt, choose ...

Average-Case Information Complexity of Learning

How many bits of information are revealed by a learning algorithm for a ...

Low complexity, low probability patterns and consequences for algorithmic probability applications

Developing new ways to estimate probabilities can be valuable for scienc...

Modeling Big Data-based Systems through Ontological Trading

One of the great challenges the information society faces is dealing wit...


We study the data-scaling of transfer learning from foundation models in...

Principles of modal and vector theory of formal intelligence systems

The paper considers the class of information systems capable of solving ...

Please sign up or login with your details

Forgot password? Click here to reset