Minimum Description Length Induction, Bayesianism, and Kolmogorov Complexity

by   Paul Vitanyi, et al.
University of Waterloo
Centrum Wiskunde & Informatica

The relationship between the Bayesian approach and the minimum description length approach is established. We sharpen and clarify the general modeling principles MDL and MML, abstracted as the ideal MDL principle and defined from Bayes's rule by means of Kolmogorov complexity. The basic condition under which the ideal principle should be applied is encapsulated as the Fundamental Inequality, which in broad terms states that the principle is valid when the data are random, relative to every contemplated hypothesis and also these hypotheses are random relative to the (universal) prior. Basically, the ideal principle states that the prior probability associated with the hypothesis should be given by the algorithmic universal probability, and the sum of the log universal probability of the model plus the log of the probability of the data given the model should be minimized. If we restrict the model class to the finite sets then application of the ideal principle turns into Kolmogorov's minimal sufficient statistic. In general we show that data compression is almost always the best strategy, both in hypothesis identification and prediction.


page 1

page 2

page 3

page 4


Extending Universal Intelligence Models with Formal Notion of Representation

Solomonoff induction is known to be universal, but incomputable. Its app...

Introduction to minimum message length inference

The aim of this manuscript is to introduce the Bayesian minimum message ...

The Optimal Choice of Hypothesis Is the Weakest, Not the Shortest

If A and B are sets such that A ⊂ B, generalisation may be understood as...

A Universal Hypercomputer

This paper describes a type of infinitary computer (a hypercomputer) cap...

Minimum Description Length Revisited

This is an up-to-date introduction to and overview of the Minimum Descri...

Optimal Work Extraction and the Minimum Description Length Principle

We discuss work extraction from classical information engines (e.g., Szi...

The whole and the parts: the MDL principle and the a-contrario framework

This work explores the connections between the Minimum Description Lengt...

Please sign up or login with your details

Forgot password? Click here to reset