The Kolmogorov complexity of a string , , is the size of the smallest program that outputs with respect to a universal prefix-free program. It is a well known fact that Kolmogorov complexity is uncomputable (see [Kol65] and [Sol64]). In fact any computable function that is not greater than is bounded by a constant. This is because for each , you can find an such that . Thus can be identified with and , so . However since , we have , thus causing a contradiction for large enough .
The authors in [BFNV05], using expanding graphs, introduced an algorithm that when given a non-random string, outputs a small list of strings of the same length containing a string with higher complexity. In [Zim16], an algorithm was presented that when given a non-random string, outputs a large list of strings of the same length where 99% of the outputted strings have higher complexity. Given a universal machine , a -short program for is a string such that and the length of is bounded by + . The authors in [BMVZ13] showed that there exists a computable function that maps every to a list of size containing a -short program for .
In this paper, we show that the approximate knowledge about the Kolmogorov complexity of a finite number of strings is equivalent to sharing a certain amount of information with the halting sequence. The more strings in the collection and the better their approximation to , the more information this collection has with the halting sequence. The mutual information between an encoding of unique numbers alongside their Kolmogorov complexity and the halting sequence is at least . Due to information non-growth laws, there is no (randomized) algorithmic means to produce information with the halting sequence.
We also provide a generalization of “Sets Have Simple Members” theorem, first seen in [EL11]
, to conditional Kolmogorov complexity and conditional algorithmic probability. The theorem states that the minimum conditional complexity, over pairs specified by an binary relation is less than the negative log of the combined conditional algorithmic probability of all pairs in the enumeration.
We use , , , , , to denote natural numbers, rational numbers, real numbers, bits , finite strings, and infinite sequences. We use and to denote the positive and non-negative elements of set . The th bit a string is . For string , . The length of a string is . The size of a set is . For and , we use iff or there is some string where . We say iff and . The self delimiting code of a string is . The encoding of (a possibly ordered) set , is .
A (discrete) measure is a function . Measure is a semi-measure iff . Measure is a probability measure iff . The support of a measure is . Probability measure is elementary if and . Elementary probability measures with are encoded by finite strings of the . For semi-measure , we say function is a test, iff .
For nonnegative real function , we use , , to denote , , and . We also use and to denote and .
We use algorithms on input programs and auxilliary inputs . is a prefix free algorithm if for all and , , does not halt or does not halt. There exists a universal prefix algorithm where for all prefix algorithm there exists a , where for all and , . As is standard, we define Kolmogorov complexity with respect to , with for , . The universal probability is defined as . By the coding theorem
. By the chain rule,.
Let be a function with a finite domain , . Then , where , is . . The complexity of general partial computable functions is defined as the length of the shortest -program that computes it. The halting sequence is the unique infinite sequence where is equal to 1 iff halts. The information that has about is .
3 Left-Total Machines
This paper uses notions of total strings and left-total machines. A string is total if all sufficiently long extensions of
will cause the universal Turing machineto halt. More formally is total if and only if there exists a finite prefix free set of strings such that , and for all , halts. Along with totality, we introduce the notion of leftness. We say is to the left of , , iff there exists a string such that and . We say the universal Turing machine is left total if for all strings , with , if halts then is total. An example of the domain of a left total machine can be seen in Figure 1. This example also illustrates the reason for using “left” in the definition.
Without loss of generality, we can assume that the universal Turing machine is left-total. We refer the readers to [Eps19b] on the explicit construction of a left-total universal Turing machine. The border sequence is the unique sequence where if then has both total and non-total extensions. The sequence is called “border” because if then is total and if , then will never halt when given as the starting input.
For total string , we define the following function as the longest running time of a program that is to the left of or extends . If and are total, then . For total string , and , let be the algorithmic weight of from programs conditioned on in time . More formally,
The term is 0 if is not total. If and are total, then .
This paper uses the notion of stochasticity, which is a part of algorithmic statistics. For a comprehensive survey of algorithmic statistics, see [VS17]. A string is stochastic if it is typical of a simple probability measure. Typicality is measured by the deficiency of randomness . The deficiency of randomness of a string , with respect to a probability measure , conditioned on auxillary string , is
The deficiency of randomness measures the difference between the length of the -Shannon-Fano code for and the shortest description of (given ). If is typical, then its measure will be small. We say a string is stochastic conditional to , for and , if there exists a program of length where , and is an elementary probability measure, and . The stochasticity measure of a string , conditional on auxilliary information is
The following lemma is from [EL11]. It states that strings that have high stochasticity measures are exotic, in that they have high mutual information with the halting sequence. Another version of the lemma can be found in [Eps19b].
For , .
The following lemma is from [Eps19a]. A variant of the same idea can found in Proposition 5 of [VS17]. It states that there is no total computable function can increase the stochasticity of a string by more than a constant factor (dependent on the complexity of the function).
Given total recursive function , .
The following lemma is from [Eps19b]. If a string is total and is not total, then . This is because the border sequence is defined by the unique sequence whose prefixes have total and non total extensions. Since is total and is not total, has total and non-total extensions. The following lemma states that if a prefix of border is simple relative to a string and its own length, then it will be a part of the common information of and .
If is total and is not, and ,
The following theorem is from [Eps19b]. It states that given two (not necessarily probabilistic) measures and with certain summation requirements, if the combined -score of elements of set is large, then there exist an element in can be identified by low -code.
Relativized to computable an with , if for some finite set , , then there exists with .
5 Uncomputability of
Corollary 1 shows that an encoding of any unique pairs has more than bits of mutual information with the halting sequence . So all such large sets are exotic.
For any finite set of natural numbers and , where , we have .
Let . Note that by the coding theorem . Let be the shortest total string with .
as there is a program that when given , and , enumerates all total strings of length and returns the first where , which we call satisfying property . This is equal to , otherwise there is a , that satisfies property . This implies that is total and satisfies property . This implies a contradiction for being the smallest total string satisfying property . The same arguments can be used if . This also implies is not total. A graphical depiction of this argument can be seen in Figure 2.
So for all , . Let , and . . Theorem 1, relativized to , gives where . So
If , then the theorem is trivially solved. So, assuming , we have . So . Therefore . So .
Any set of unique pairs has .
6 Exotic Binary Relations
For -programs that enumerate a (potentially infinite) binary relation and total string , we use to denote the finite binary relation enumerated by in steps. We use to denote the entire binary relation enumerated by .
For -program that enumerates a binary relation, with
Let be the shortest total string where . We have the inequality because there is a program that when given , , and , can enumerate all total strings of length and all pairs , and return the first total string where , which we call satisfying property . This string is unique, otherwise there exists a string , which satisfies property . If , then is total and satisfies property , contradicting the definition of being the shortest total string satisfying property . Similar reasoning can be used for when . Therefore , and is unique. Figure 2 illustrates this point.
Let , be the program and elementary probability measure that minimize the stochasticity of conditional on , , where , and
Let be an elementary probability measure equal to conditioned on the largest set of programs that enumerate binary relations where , which we call satisfying property . Thus , where . Let , , with , where is helper code of size . Thus which implies . Let . So
Let , which is finite. Let
be a set of random vectors, indexed by, each of size . The number is a constant solely dependent on to be determined later. Each element of the vector is chosen with probability , and is chosen with probability . Let , be a nonnegative function over strings, parameterized by sets of strings , each of size , each indexed by a string . For an enumerative program , , if there exists where . Otherwise . So, using the fact for ,
Thus there exists a collection of sets , indexed by , where . This collection can be found using brute search given , , and , with .
There exists , and where . Otherwise , and for proper choice of , solely dependent on , we have
causing a contradiction. We roll into the additive constants for the rest of the theorem. So , and there exists an , where . So
Equation 3 is due to the fact that is a program (conditioned on ). So its conditional complexity is not more than its length. Equation 4 is due to Equation 2. Equation 5 is due to Lemma 1. Equation 6 is due to Lemma 3. Equation 7 is to the inequality .
For finite binary relation , with , .
For partial computable function with ,
- [BFNV05] H. Buhrman, L. Fortnow, I. Newman, and N. Vereshchagin. Increasing kolmogorov complexity. In STACS 2005, pages 412–421, 2005.
- [BMVZ13] B. Bauwens, A. Makhlin, N. Vereshchagin, and M. Zimand. Short lists with short programs in short time. In 2013 IEEE Conference on Computational Complexity, pages 98–108, 2013.
- [EL11] Samuel Epstein and Leonid Levin. On sets of high complexity strings. CoRR, abs/1107.1458, 2011.
- [Eps19a] S. Epstein. On the Complexity of Completing Binary Predicates. arXiv e-prints, page arXiv:1907.04776, 2019.
- [Eps19b] Samuel Epstein. All sampling methods produce outliers. CoRR, abs/1304.3872, 2019.
- [Kol65] A. N. Kolmogorov. Three approaches to the quantitative definition of information. Problems in Information Transmission, 1:1–7, 1965.
- [Sol64] R. J. Solomonoff. A Formal Theory of Inductive Inference, Part l. Information and Control, 7:1–22, 1964.
- [VS17] Nikolay K. Vereshchagin and Alexander Shen. Algorithmic statistics: Forty years later. In Computability and Complexity, pages 669–737, 2017.
- [Zim16] Marius Zimand. List approximation for increasing kolmogorov complexity. CoRR, abs/1609.05984, 2016.