Emergence of order in random languages

1 Partition functions

A CFG in Chomsky Normal form is defined by two types of rules: $a \to b c$ , where one hidden symbol becomes two hidden symbols, and $a \to B$ , where a hidden symbol becomes an observable symbol. In a weighted CFG (WCFG), we assign weights $M_{a b c}$ and $O_{a B}$ , respectively, to these rules. With these weights we build a weight for an entire derivation tree as follows. We write $σ$ for the hidden variables, and $o$ for the observables. These are indexed by their positions on a tree. Write $T$ for the topology of the tree, namely the identity (observable or hidden) of each node. We write $Ω_{T}$ for the set of internal factors, i.e. factors of the form $a \to b c$ , and $\partial Ω_{T}$ for the boundary factors, i.e. those associated to $a \to B$ rules. The number of boundary factors is written $ℓ_{T}$ , which is also the number of leaves. Since derivations are trees, the number of internal factors is $ℓ_{T} - 1$ .

Consider a derivation tree, such as that depicted in Figure 1a. In a weighted context-free grammar, a tree of topology $T$ with hidden variables $σ_{i}$ and observables $o_{t}$ has a weight

W ({σ_{i}, o_{t}} | T, G) = \prod α \in Ω_{T} M_{σ_{α_{1}} σ_{α_{2}} σ_{α_{3}}} \prod α \in \partial Ω_{T} O_{σ_{α_{1}} o_{α_{2}}},

(1)

where each $α = (α_{1}, α_{2}, α_{3})$ is a factor in the order $σ_{α_{1}} \to σ_{α_{2}} σ_{α_{3}}$

. This defines a conditional probability measure on configurations

P ({σ_{i}, o_{t}} | T, G) = \frac{W ({σ_{i}, o_{t}} | T, G)}{Z (T, G)}

(2)

where

Z (T, G) = \sum {σ_{i}, o_{t}} W ({σ_{i}, o_{t}} | T, G)

(3)

We will additionally add a weight depending only on the size of the tree, $P (T | G) = W_{t r e e} (T) / Z_{t r e e}$ with $W_{t r e e} (T) = p^{| \partial Ω_{T} |} (1 - p)^{| Ω_{T} |}$ , where $p$ is the emission probability, which controls the size of trees ¹¹1 $Z_{t r e e} = 2 p / (1 + | 2 p - 1 |) = 1$ for $p > 1 / 2$ .. The tree-averaged partition function is then

Z (G; ℓ) = \sum T, ℓ (T) = ℓ P (T | G) Z (T, G)

(4)

as a function of the number of leaves $ℓ$ , and the partition function for $m$ sentences of total length $ℓ$ is

Z (G; m, ℓ) = \sum {ℓ_{i}}, \sum_{i = 1}^{m} ℓ_{i} = ℓ \prod i Z (G; ℓ_{i})

(5)

We have $\sum_{i} | \partial Ω_{T_{i}} | = \sum ℓ_{i} = ℓ$ , and $\sum_{i} | Ω_{T_{i}} | = \sum (2 ℓ_{i} - 1) = 2 ℓ - m$ , so that $Z_{t r e e} \equiv \prod_{i} P (T_{i} | G) = p^{ℓ} (1 - p)^{2 ℓ - m} Z_{t r e e}^{- m}$ just gives a trivial factor. For now we suppress dependence on $m$ and $ℓ$ . Note that $Z (G; m, ℓ)$ is the weight for the grand canonical partition function $\sum_{m, ℓ \geq 0} Z (G; m, ℓ)$ ; we will, however, work at fixed $m$ and $ℓ$ .

2 Energy, Entropy, and Order Parameters

It is convenient to add some auxiliary parameters in order to extract additional observables. First, we note that (1) can be written as

W ({σ_{i}, o_{t}} | T, G) = \prod a, b, c M_{a b c}^{π_{a b c}} \prod a, B O_{a B}^{ρ_{a B}}

(6)

where $π_{a b c} (σ)$ is the usage frequency of rule $a \to b c$ and $ρ_{a B} (σ, o)$ is the usage frequency of $a \to B$ . Adding external fields $h_{a b c}$ and $k_{a B}$ let us define the energy of a configuration as

E = - \sum a, b, c π_{a b c} [h_{a b c} + log M_{a b c}] - \sum a, B ρ_{a B} [k_{a B} + log O_{a B}]

(7)

Then we can generalize (3) to

Z_{h, k} (T, G) = \sum {σ_{i}, o_{t}} e^{- β E}

(8)

where we added a bias $β$ . The original ensemble is recovered for $β = 1$ and $h = k = 0$ . We see that the average energy is

⟨ E ⟩ = - \frac{\partial log Z}{\partial β}

(9)

and it is natural to define the entropy of the grammar as $S = β ⟨ E ⟩ + log Z$ .

In [1] it was argued that a natural order parameter for WCFGs is one that measures the extent to which rules are applied uniformly: if all rules $a \to b c$ and $a \to B$ have the same weight, the grammar carries no information, and sentences will be indistinguishable from noise. To measure order in the deep grammar, define first

(10)

averaged over all interior vertices $α$ , and averaged over derivations. The normalization $N_{M} = ℓ - m$ for $Z$ . A spin-glass order parameter specific to deep structure is

Q_{2} \equiv ¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯ \sum a, b, c Q_{a b c}^{2} = \frac{N^{5} (N^{2} - 1)}{N_{M}^{2}} (q_{0} - q_{1})

(11)

where

q_{0} = ¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯ ⟨ π_{a b c} ⟩^{2}, q_{1} = ¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯ ⟨ π_{a b c} ⟩ ⟨ π_{a b^{'} c^{'}} ⟩,

(12)

with $b^{'} \neq b, c^{'} \neq c$ . Here we used the fact that when $h = 0, k = 0$ , the permutation symmetry is restored upon disorder averaging. We see that

⟨ π_{a b c} ⟩ = \frac{1}{β} \frac{\partial log Z}{\partial h_{a b c}}

(13)

so $q_{0}$ and $q_{1}$ can be obtained from derivatives of $¯ ¯¯¯¯¯¯¯¯¯¯ ¯ log Z$ with respect to the field.

3 Diagrammatic formulation

We expect that universal properties of weighted context-free languages are contained in the behavior of $¯ ¯¯¯¯¯¯¯¯¯¯ ¯ log Z$ when $ℓ$ becomes large, where $¯ ¯¯¯ ¯ \cdot$ is an average over grammars. In order to compute this object, we find it convenient to move to an alternative, particle, representation. In particular, we seek a model whose diagrammatic expansion gives the derivation trees we seek to count, with the appropriate weights. This technique has been widely used in the study of 2D gravity [6], and facilitated Kazakov’s solution of the Ising model on random surfaces [7, 8]. Later, it was shown in a simpler setting that this technique could be used to easily obtain results for spin models on random graphs [9, 10].

Figure 3: Feynman rules for weighted context-free grammars. Elements of Feynman diagram expansion (top row) with weights (middle row). (a-b) $M -$ interaction, (c-d) Root source, (e-f) $O$ source, (g-h): Nonzero propagators. Labels $a, a^{'}, b, c$ are colour indices, $\in {1, \dots, N}$ . All propagators are diagonal in colour.

We begin with a simplified model. Consider the formal integral

W = | G |^{- N / 2} \int D U e^{- \frac{1}{2} \sum_{i, j} U_{i} G_{i j}^{- 1} U_{j}} e^{ξ \sum_{i} U_{i} + η \sum_{i, j, k} U_{i} U_{j} U_{k} M_{i j k}},

(14)

where the measure is normalized such that $| G |^{- N / 2} \int D U exp (- \frac{1}{2} \sum_{i, j} U_{i} G_{i j}^{- 1} U_{j}) = 1$ . Strictly speaking, the integral is only defined by its perturbative expansion in $η$ ; convergence requires that the real part of $G^{- 1}$ be positive-definite. This expansion generates Feynman diagrams with cubic vertices. Each vertex gets a factor $η M_{i j k}$ , and the expansion with respect to $ξ$ generates sources. By Wick’s theorem, each edge gets a factor $G_{i j}$ , the propagator. The coefficient in this expansion of $ξ^{m} η^{k}$ thus counts all such diagrams, possibly disconnected, with $k$ vertices and $m$ sources, times an inverse symmetry factor [11] ²²2Such factors appear when diagrams, including all their colour indices, have nontrivial symmetries, like reflections. In the disordered case where $M_{a b c}$ depends on all indices, these symmetry factors will not play a role since typical connected graphs will have no symmetries..

This is a skeleton of what we need to count derivation trees, but there are several elements missing: first, $W$ includes all graphs with cubic vertices, not only trees. Second, even if we could restrict the sum to trees, there is nothing in $W$ to distinguish leaves from roots, or to distinguish the left and right branches from a given hidden node.

Figure 4: Feynman diagram corresponding to derivation tree in Figure 1a. Alphabet of hidden symbols is $χ_{d} = (S, N P, V P, D e t, N, V, P, P P)$ and alphabet of surface symbols is $χ_{s} = (t h e, b e a r, w a l k e d, i n t o, c a v e)$ . Vertices are represented by $\land$ with heads at the tip. The diagram has a weight $2 h ξ^{6} η^{5} g^{11} M_{123} M_{245}^{2} M_{368} M_{872} O_{41}^{2} O_{52} O_{63} O_{74} O_{55}$ .

One solution to these problems is to use matrices as the integration variables, because their diagrammatic expansion can be arranged to give planar diagrams in an appropriate large $N$ limit [12, 13]. However, for our problem this is overkill. Instead, we will consider a theory of complex scalar fields with colour indices, equivalent to a complex matrix model with matrices of size 1x1. We will have two scalar fields $L_{a}$ and $R_{a}$ , with colour indices $a = 1 \dots N$ . Consider

F (G) = \int D L \int D R e^{- \frac{1}{g} \sum_{a} [L_{a} L_{a}^{†} + R_{a} R_{a}^{†}]} e^{I}

(15)

where $^{†}$ denotes complex conjugate and

I = ζ h (L_{1} + R_{1}) + ξ \sum a O_{a} (L_{a}^{†} + R_{a}^{†}) + η \sum a, b, c M_{a b c} (L_{a}^{†} + R_{a}^{†}) L_{b} R_{c} .

(16)

The measure $D L = \prod_{a} d R e [L_{a}] d I m [L_{a}] / (π g)$ is normalized such that $\int D L e^{- \frac{1}{g} \sum_{a} L_{a} L_{a}^{†}} = 1$ , and similarly for $R$ .

The propagator is diagonal in the colour indices $a = 1 \dots N$ and for each $a$ is such that $⟨ L_{a}^{†} L_{a} ⟩ = g$ ; that is, the Feynman rules are as shown in Figure 3. The diagram corresponding to Figure 1a is shown in Figure 4. We claim that, apart from accidental symmetry factors,

Z (G; m, ℓ) = m! \oint^{'} \frac{d ζ}{ζ^{1 + m}} \oint^{'} \frac{d ξ}{ξ^{1 + ℓ}} \oint^{'} \frac{d η}{η^{1 + ℓ - m}} F (G),

(17)

where $\oint^{'} = \oint / (2 π i)$ . The proof is as follows.

The perturbative expansion with respect to $η$ generates cubic vertices with distinguished heads and $L$ and $R$ ends; the vertices can be placed on the plane such that that their heads point up. The expansion and contour integral for $ξ$ generate $ℓ$ sources of $L^{†}$ and $R^{†}$ , which are the leaves. The expansion and contour integral for $ζ$ generate $m$ sources of $L_{1}$ and $R_{1}$ , which are the roots. An $L_{a}$ can only be connected to a $L_{a}^{†}$ , and a $R_{a}$ can only be connected to a $R_{a}^{†}$ . We can orient all edges from $L_{a} \to L_{a}^{†}$ and $R_{a} \to R_{a}^{†}$ , and similarly for the half-edges in the cubic vertices, flowing from head to L and R branches. Then, from each root, we can define paths by following arrows; any path we take will go through some number of cubic vertices and end in a leaf. Therefore there are $m$ connected components, one for each root. Considered as a graph, we can count the number of edges as follows: each source generates half an edge, and each cubic vertex generates $3 / 2$ edges. The total number of edges is $\frac{1}{2} (m + 3 (ℓ - m) + ℓ) = 2 ℓ - m$ . The difference between the number of vertices and the number of edges is $(m + ℓ + ℓ - m) - (2 ℓ - m) = m$ , which is the number of connected components. Therefore the graph is a forest.

We thus generate a forest of $m$ planar, rooted, trees, with $ℓ$ leaves in total. The weight of each diagram is

h^{m} g^{2 ℓ - m} \prod a, b, c M_{a b c}^{π_{a b c}} \prod a B O_{a B}^{ρ_{a B}}

(18)

where $π_{a b c}$ is the usage frequency of rule $a \to b c$ and $ρ_{a B}$ is the usage frequency of $a \to B$ . This expansion counts diagrams with a degeneracy of $2^{m}$ since each tree root can be either an $L$ or an $R$ . In the expansion, the different connected components are not ordered. We would like to distinguish forests by the order of the trees, so we multiply the result by $m!$ . Choosing $g$ and $h$ such that

(2 h)^{m} g^{2 ℓ - m} = p^{ℓ} (1 - p)^{2 ℓ - m} Z_{t r e e}^{- m}

(19)

for all $ℓ$ and $m$ we have our result.

The virtue of working with (17) is that when $ℓ \to \infty, m / ℓ = α =$ constant, the leading behavior can be extracted by a saddle-point analysis [14] ³³3This can be seen explicitly by considering the rescaling $L = ℓ^{1 / 2} L^{'}, R = ℓ^{1 / 2} R^{'}, η = ℓ^{- 1 / 2} η^{'}, ξ = ℓ^{1 / 2} ξ^{'}, ζ = ℓ^{1 / 2} ζ^{'}$ . . There is one subtlety. The integration variables are the real and imaginary parts of $L$ and $R$ , and the saddle-point equations should be taken with respect to these parts. The solutions to $R e [L_{a}]$ and $I m [L_{a}]$ , which may be complex, are then added to produce $L_{a} = R e [L_{a}] + i I m [L_{a}]$ , and similarly for $R$ . By linearity, this is equivalent to taking saddle-point equations with respect to $L_{a}$ and $L_{a}^{†}$ , and treating $L_{a}$ and $L_{a}^{†}$ as independent. It is convenient to write $H_{a} = L_{a}^{†} + R_{a}^{†}$ . The saddle-point equations are

$L_{a}$	$= g ξ O_{a} + g η \sum b, c M_{a b c} L_{b} R_{c}$	(20)
$L_{a}^{†}$	$= g h ζ δ_{a 1} + g η \sum a^{'}, b, c M_{a^{'} b c} H_{a} δ_{a b} R_{c}$	(21)
$R_{a}$	$= g ξ O_{a} + g η \sum b, c M_{a b c} L_{b} R_{c}$	(22)
$R_{a}^{†}$	$= g h ζ δ_{a 1} + g η \sum a^{'}, b, c M_{a^{'} b c} H_{a} L_{b} δ_{a c}$	(23)
$ℓ - m$	$= η \sum a, b, c M_{a b c} H_{a} L_{b} R_{c}$	(24)
$ℓ$	$= ξ \sum a O_{a} H_{a}$	(25)
$m$	$= ζ h (L_{1} + R_{1})$	(26)

for all $a$ .

(20),(21) and their pairs (22),(23) have an interpretation as recursion equations, which are equivalent to the saddle-point limit of Tutte recursion relations or loop equations in related contexts [15, 16]. They are also related to self-consistent equations derived for spin glasses on trees ⁴⁴4For example, $L_{a}$ is analagous to what is called $ϕ ({σ^{a}})$ in [9] and $g_{n} ({σ_{a}})$ in [17]. In these works, $ϕ$ and $g_{n}$ are functions of $n$ Ising variables, so that they can take $2 n$ different values. Below, we will replicate the $L_{a}$ so that they take $n N$ different values; $N = 2$ is the Ising case.. Indeed, any $L_{a}$ node can either propagate into $L_{a}^{†}$ and become a leaf, with weight $g ξ O_{a}$ , or propagate into $L_{a}^{†}$ and become another branch, with weight $g η \sum_{b, c} M_{a b c} L_{b} R_{c}$ , including all possibilities. This gives (20). Similarly, any $L_{a}^{†}$ node is either the child of a root, with weight $g ζ h δ_{b 1}$ , or the child of a branch, with weight $g η \sum_{a^{'}, a, c} M_{a^{'} a c} (L_{a}^{†} + R_{a}^{†}) R_{c}$ , including all possibilities. This gives (21).

For specific grammars, (20)-(26) can be explicitly analyzed ⁵⁵5After writing these equations in terms of real variables, these take the form of ‘context-free schema.’ See section VII.6.1 in [18].. Our interest, however, is in extracting the behavior of typical grammars in the case when $N$ is large. For this we need to choose an ensemble.

4 Grammar ensembles

Figure 5: Distribution of individual grammar weights in (a) Random Language model (RLM) and (b) Gaussian model for ${~ ϵ}_{d}$ as indicated. The regime of unphysical negative weights is shaded in the Gaussian model.

We will consider two models. In [1] it was argued that a generic model will have lognormally distributed weights, viz.,

P_{G} (M, O)

\equiv Z_{G}^{- 1} J e^{- ϵ_{d} s_{d}} e^{- ϵ_{s} s_{s}}

(27)

where the deep and surface sparsities $s_{d}$ and $s_{s}$ are defined by

s_{d} = \frac{1}{N^{3}} \sum a, b, c {log}^{2} [\frac{M_{a b c}}{¯ ¯¯¯¯ ¯ M}], s_{s} = \frac{1}{N T} \sum a, B {log}^{2} [\frac{O_{a B}}{¯ ¯¯ ¯ O}]

(28)

and $J = e^{- \sum_{a, b, c} log M_{a b c} - \sum_{a, B} log O_{a B}}$ . Here $¯ ¯¯¯¯ ¯ M = 1 / N^{2}$ and $¯ ¯¯ ¯ O = 1 / T$ . A plot of the weights for a range of ${~ ϵ}_{d} = ϵ_{d} / N^{3}$ is shown in Fig.5a. It is straightforward to show that $ϵ_{d}$ and $ϵ_{s}$ satisfy

_{d} = (2 {~ ϵ}_{d})^{- 1},_{s} = (2 {~ ϵ}_{s})^{- 1} .

(29)

where $¯ ¯¯¯ ¯ \cdot$ denotes a grammar average and ${~ ϵ}_{s} = ϵ_{s} / (N T)$ . A small ‘deep temperature’ $ϵ_{d}$ corresponds to a large deep sparsity. The model (27) was called in [1] the Random Language Model (RLM).

It was shown in [1] that the RLM shows two phases, depending on the value of ${~ ϵ}_{d}$ , plus logarithmic corrections. More precisely, Shannon entropies appear to collapse with respect to ${~ ϵ}_{d} {log}^{k} N$ , where $k = 1$ or $k = 2$ depending on the quantity considered. For ${~ ϵ}_{d} {log}^{k} N ≳ 1$ , Shannon entropies are independent of the deep temperature $ϵ_{d}$ , and take maximal values, indicating that the grammar does not carry information: despite strictly following the rules of a WCFG, sentences are indistinguishable from random noise. For smaller $ϵ_{d}$ , entropies drop, and the grammar carries nontrivial information. It is our goal to extract this transition from (17).

It will turn out to be much simpler to consider an alternative model, where the weights $M_{a b c}$ and $O_{a B}$

are Gaussian, rather than lognormal; matching the mean and variance to those of the RLM we can again use the quantities

¯ ¯¯¯¯ ¯ M

and

{~ ϵ}_{d}

, and similarly for

O_{a B}

. The distribution is plotted in Fig.5b. This model has the unphysical feature that weights have a negative tail; naively, we could imagine that this would be unimportant, since the largest weights are most important, but we will have to revise this statement later. We call this the Gaussian model (GM).

We wish to compute

¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯ log Z (G) =_{n = 0} = {\frac{\partial}{\partial n} ∣ ∣ ∣}_{n = 0} ¯ ¯¯¯¯¯¯¯¯¯¯¯¯ ¯ Z (G)^{n},

(30)

where we used the replica method [19]. The fields $ζ, ξ, η, L$ , and $R$ are all replicated, adding an index $i = 1 \dots n$ . To compute $¯ ¯¯¯¯¯¯¯¯¯¯¯ ¯ F (G)^{n}$ we need the grammar average

	$A (ξ, η, L, R)$	$=^{ξ \sum_{i, a} O_{a} H_{a}^{i} + η \sum_{i, a, b, c} M_{a b c} H_{a}^{i} L_{b}^{i} R_{c}^{i}}$
		$= \prod a, B^{ξ O_{a B} \sum_{i} H_{a}^{i}} \prod a, b, c^{η M_{a b c} \sum_{i} H_{a}^{i} L_{b}^{i} R_{c}^{i}}$		(31)

Write $x_{a b c} = \sum_{i} η_{i} H_{a}^{i} L_{b}^{i} R_{c}^{i}$ . In the RLM the grammar averages over $M$ are of the form ( $m = log M_{a b c} / ¯ ¯¯¯¯ ¯ M$ )

	$\sqrt{{~ ϵ}_{d} / π} \int d m e^{- {~ ϵ}_{d} m^{2}} e^{e^{m} ¯ ¯¯¯ ¯ M x}$	$= \sqrt{{~ ϵ}_{d} / π} \int d m e^{- {~ ϵ}_{d} m^{2}} \sum q \geq 0 \frac{{¯ ¯¯¯¯ ¯ M}^{q} x^{q}}{q!} e^{q m}$		(32)
		$= \sum q \geq 0 \frac{{¯ ¯¯¯¯ ¯ M}^{q} x^{q}}{q!} e^{q^{2} / (4 {~ ϵ}_{d})}$		(33)

A term of order $q$ corresponds to the rule appearing $q$ times. We are interested in a transition due to patterns of repeated rule application between sentences, rather than inside them (this would correspond to a transition deeper in the ordered phase). Therefore a priori we expect that we only need connected terms up to a small finite order $q_{*}$ . Note that at order $q_{*}$ , terms involving $q_{*}$ different replicas will be present. Resumming gives

\sum q \geq 0 \frac{{¯ ¯¯¯¯ ¯ M}^{q} x^{q}}{q!} e^{q^{2} / (4 {~ ϵ}_{d})} = exp (\sum q \geq 1 a_{q} {¯ ¯¯¯¯ ¯ M}^{q} x^{q})

(34)

with $a_{1} = e^{1 / (4 {~ ϵ}_{d})}, a_{2} = \frac{1}{2} (e^{4 / (4 {~ ϵ}_{d})} - e^{2 / (4 {~ ϵ}_{d})}),$ etc. From this divergent sum we only need to retain terms up to order $q = n (ℓ - m)$ , since the integration over $η$ will retain only $ℓ - m$ vertices for each replica.

For practical reasons exact calculations are limited to $q_{*} = 2$ . In this case, we consider all derivations in which a rule can appear at most twice in one derivation tree. Note that rules can still appear arbitrarily many times in the set of $n$ replicas and $m$ sentences. Keeping terms to $q_{*} = 2$ is equivalent to letting the $M$

be drawn from a Gaussian distribution. For appropriate choice of mean and variance, we can thus fix the GM to be equal to the RLM to this order. In the remainder of this work, we will first find the exact solution of the GM, and then discuss its extension to the RLM.

4.1 Gaussian model

Applying the same arguments to the integral over $O$ , we have for the GM

	$A (ξ, η, L, R)$	$= \prod a, B e^{b_{1} ¯ ¯¯ ¯ O \sum_{i} ξ_{i} H_{a}^{i} + b_{2} {¯ ¯¯ ¯ O}^{2} (\sum_{i} ξ_{i} H_{a}^{i})^{2}} \prod a, b, c e^{a_{1} ¯ ¯¯¯ ¯ M \sum_{i} η_{i} H_{a}^{i} L_{b}^{i} R_{c}^{i} + a_{2} {¯ ¯¯¯ ¯ M}^{2} (\sum_{i} η_{i} H_{a}^{i} L_{b}^{i} R_{c}^{i})^{2}}$
		$= e^{b_{1} N T ¯ ¯¯ ¯ O \sum_{i} ξ_{i} H_{}^{i} + b_{2} T {¯ ¯¯ ¯ O}^{2} \sum_{i, j} ξ_{i} ξ_{j} Q_{H}^{i j} + a_{1} N^{3} ¯ ¯¯¯ ¯ M \sum_{i} η_{i} H_{}^{i} L_{}^{i} R_{}^{i} + a_{2} {¯ ¯¯¯ ¯ M}^{2} \sum_{i, j} η_{i} η_{j} Q_{H}^{i j} Q_{L}^{i j} Q_{R}^{i j}}$		(35)

where we introduced ‘magnetization’ vectors and overlap matrices

	$H_{}^{i} = \frac{1}{N} \sum a H_{a}^{i}, L_{}^{i} = \frac{1}{N} \sum a L_{a}^{i}, R_{*}^{i} = \frac{1}{N} \sum a R_{a}^{i},$		(36)
	$Q_{H}^{i j} = \sum a H_{a}^{i} H_{a}^{j}, Q_{L}^{i j} = \sum a L_{a}^{i} L_{a}^{j}, Q_{R}^{i j} = \sum a R_{a}^{i} R_{a}^{j},$		(37)

and $b_{1} = e^{1 / (4 {~ ϵ}_{s})}, b_{2} = \frac{1}{2} (e^{4 / (4 {~ ϵ}_{s})} - e^{2 / (4 {~ ϵ}_{s})})$ . (Recall that $H_{a} = L_{a}^{†} + R_{a}^{†}$ .). Assembling the above results we find that for the GM,

	$¯ ¯¯¯¯¯¯¯¯¯¯¯¯ ¯ Z (G)^{n}$	$= \prod i [m! \oint^{'} \frac{d ζ_{i}}{ζ_{i}^{1 + m}} \oint^{'} \frac{d ξ_{i}}{ξ_{i}^{1 + ℓ}} \oint^{'} \frac{d η_{i}}{} η_{i}^{1 + ℓ - m} \int D L^{i} \int D R^{i} e^{- \frac{1}{g} \sum_{a} [L_{a}^{i} L_{a}^{i}^{†} + R_{a}^{i} R_{a}^{i}^{†}]}]$
		$e^{\sum_{i} ζ_{i} h (L_{1}^{i} + R_{1}^{i})} A (ξ, η, L, R)$

This is now in the form amenable to standard treatment by replicas: the overlap matrices can be introduced as new parameters, and the original variables can be integrated out. We notice that the colour indices play the role usually played by spatial indices in spin glasses [19]. The surprising result is that the model can be exactly integrated, without even making an ansatz on the replica structure, and without taking the large $ℓ$ limit. This integrability can be traced to the fact that the overlap matrices depend only on the real and imaginary parts of $L$ in the canonical way, i.e. through $L_{a}^{i} = R e [L_{a}^{i}] + i I m [L_{a}^{i}]$ . This gives rise to a symplectic structure that simplifies the integration over these variables. The derivation is sketched in Appendix A. The final result is

	$¯ ¯¯¯¯¯¯¯¯¯¯ ¯ log Z$	$= m log h + (2 ℓ - m) log g + S_{ℓ, m} + ℓ log (T ¯ ¯¯ ¯ O b_{1}) + ℓ_{0} log (N^{2} ¯ ¯¯¯¯ ¯ M a_{1})$		(38)
		$+ ℓ f (x_{s}) + ℓ_{0} f (x_{d})$

with $ℓ_{0} = ℓ - m$ ,

(39)

and

S_{ℓ, m} = log ℓ_{0}! - log ℓ! + log (1 + ℓ + ℓ_{0})! - log (1 + 2 ℓ_{0})!

(40)

Here $x_{d} = N^{- 3 / 2} \sqrt{e^{1 / (2 {~ ϵ}_{d})} - 1}$ and $x_{s} = (N T)^{- 1 / 2} \sqrt{e^{1 / (2 {~ ϵ}_{s})} - 1}$ .

The elements of $¯ ¯¯¯¯¯¯¯¯¯¯ ¯ log Z$ are as follows. Those involving $h$ and $g$ are precisely those required from 18. $S_{ℓ, m}$ is entropic. Terms with $b_{j}$ and $a_{j}$ are energetic, since they depend on the grammar weight distribution. The function $f (x)$ , which can written in terms of hypergeometric functions, is plotted in Fig.6. It develops an imaginary part for $x \sim O (1)$ , indicating that the unphysical negative probability states are becoming important. For large $N$ the condition $x_{d} ≲ O (1)$ is equivalent to

{~ ϵ}_{d} log N ≳ 1 / 6,

(41)

and similarly the condition $x_{s} ≲ O (1)$ is equivalent to ${~ ϵ}_{s} log N T ≳ 1 / 2$ . These inequalities fix the regime in which the GM is physical.

4.2 Rlm

We now return to the full model. As discussed above, we cannot obtain the exact solution; however, since a saddle-point method is justified for large $ℓ$ , we can consider different ansatze on the form of the solution. Two are natural: (i) the colour-symmetric ansatz $L_{a}^{i} = L^{i}, R_{a}^{i} = R^{i}$ $\forall a$ , and the (ii) replica-symmetric ansatz $L_{a}^{i} = L_{a}, R_{a}^{i} = R_{a}$ $\forall i$ . After some calculations similar to those for the GM, we eventually find for either (i) or (ii) the same form (38), except that $f \equiv 0$ . Besides the ansatze on the form of $L$ and $R$ , we assume that $ℓ$ is large and that the replica limit $n \to 0$ can be taken perturbatively, i.e. keeping terms $O (n)$ in the action, as in the usual approach [19] ⁶⁶6Consistency with the GM suggests that we should be able to recover (38) with $f \equiv 0$ for that model, without necessarily taking the limit ${~ ϵ}_{d} \to \infty, {~ ϵ}_{s} \to \infty$ , in which case the $f$ terms are trivially unimportant. Indeed, if instead of taking $n \to 0$ in (49), we look for a saddle-point with large $ℓ$ , the saddle-point perturbative in $n$ gives exactly (38) with $f \equiv 0$ . This indicates that the function $f$ is nonperturbative in the replica limit.. We now analyze (38) with $f \equiv 0$ , with the understanding that this holds in the replica-symmetric regime, whose range of validity is to be determined.

It is convenient to separate $F = - ¯ ¯¯¯¯¯¯¯¯¯¯ ¯ log Z$ into its entropic and energetic contributions. This can be done exactly because the RLM has a scaling symmetry when the bias $β$ is included. Indeed, it is not hard to show that the partition function satisfies the scaling property $^{n} (β, {~ ϵ}_{d}, ¯ ¯¯¯¯ ¯ M, {~ ϵ}_{s}, ¯ ¯¯ ¯ O) =^{n} (1, {~ ϵ}_{d} / β^{2}, {¯ ¯¯¯¯ ¯ M}^{β}, {~ ϵ}_{s} / β^{2}, {¯ ¯¯ ¯ O}^{β})$ (in abuse of earlier notation). The $β -$ dependent part of $F$ is

F_{β} = - β ℓ log ¯ ¯¯ ¯ O - β ℓ_{0} log ¯ ¯¯¯¯ ¯ M - \frac{β^{2} ℓ}{4 {~ ϵ}_{s}} - \frac{β^{2} ℓ_{0}}{4 {~ ϵ}_{d}}

(42)

so that at $β = 1$ , $E = - ℓ log ¯ ¯¯ ¯ O - ℓ_{0} log ¯ ¯¯¯¯ ¯ M - \frac{ℓ}{2 {~ ϵ}_{s}} - \frac{ℓ_{0}}{2 {~ ϵ}_{d}}$ and the replica-symmetric entropy is

S_{R S} = ℓ_{0} log (g N^{2} / h) + ℓ log (g T h) - \frac{ℓ}{4 {~ ϵ}_{s}} - \frac{ℓ_{0}}{4 {~ ϵ}_{d}} + S_{ℓ, m}

(43)

The entropy cannot be negative; this gives necessary, though perhaps not sufficient, conditions on the regime where the replica-symmetric ansatz is applicable. For simplicity, consider the case $ℓ \to \infty, m / ℓ = α ≪ 1$ ( $ℓ / m$ is the typical length of a tree; we let it be large). Then one can determine that $S_{ℓ, m} = O (α ℓ)$ and, for the simulated case where the emission probability is close to $p = 1 / 2$ , $g = h \approx 1 / \sqrt{8}$ . The condition $S_{R S} > 0$ is approximately equivalent to

\frac{1}{4 {~ ϵ}_{d}} + \frac{1}{4 {~ ϵ}_{s}} ≲ log (N^{2} T / 8)

(44)

Our main concern is the emergence of deep structure, which does not depend on what happens at the surface of the tree. In the limit ${~ ϵ}_{s} \to \infty$ , this becomes ${~ ϵ}_{d} log (N^{2} T / 8) ≳ 1 / 4$ , very similar to the regime in which the GM is physical, 41.

4.3 Order parameter

Finally, we return to the order parameter $Q_{2}$

that measures deep structure. Let us first give a heuristic derivation of its value in the RLM. We need to compute

q_{0} = ¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯ ⟨ π_{a b c} ⟩^{2}

and

q_{1} = ¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯ ⟨ π_{a b c} ⟩ ⟨ π_{a b^{'} c^{'}} ⟩

, where

π_{a b c}

is the occupancy of the rule

a \to b c

. Using (7) and (13) in the diagrammatic representation, one can see that

π_{a b c} = e^{h_{a b c}} η M_{a b c} H_{a} L_{b} R_{c}

. The

π_{a b c}

satisfy the sum rule

\sum_{a b c} π_{a b c} = ℓ_{0}

and so have a mean value

¯ ¯ ¯ π = ℓ_{0} / N^{3}

. The occupancies are positively correlated with the grammar weights, since rules with higher weights are sampled more frequently. A crude estimate is then

⟨ π_{a b c} ⟩ / ¯ ¯ ¯ π \sim M_{a b c} / (¯ ¯¯¯¯ ¯ M a_{1})

(the mean value of a weight is

¯ ¯¯¯¯ ¯ M a_{1}

). This leads to the estimate

	$Q_{2}$	$\sim \frac{N^{5} (N^{2} - 1)}{ℓ_{0}^{2}} \frac{ℓ_{0}^{2}}{N^{6} {¯ ¯¯¯¯ ¯ M}^{2} a_{1}^{2}} ¯ ¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯ ¯ M_{a b c}^{2} - M_{a b c} M_{a b^{'} c^{'}}$
		$= \frac{N^{2} - 1}{N} (e^{1 / (2 {~ ϵ}_{d})} - 1),$		(45)

which indicates that order increases as $ϵ_{d}$ is lowered simply because the weight variance increases. $Q_{2}$ can be computed more precisely using replicas. After a long computation, assuming the replica symmetric ansatz and taking large $ℓ$ , one eventually finds exactly the same result, (45). Thus this simple expression is in fact the genuine replica symmetric result; it is plotted in Fig.2, where it is compared with numerical data ⁷⁷7These data have been obtained by the same methods as described in [1]. Here we have simulated many more samples ( $ℓ \sim 10^{5}$ compared to $ℓ \sim 10^{3}$ in that work) in order to resolve the large $ϵ_{d}$ part of the curve.. In the large $ℓ$ limit, it matches quantitatively, without fitting parameters, above ${~ ϵ}_{d} log N ≳ 1$ . For smaller ${~ ϵ}_{d}$ , the data asymptote, as they must, while the replica-symmetric prediction diverges.

5 Conclusion

We showed that the partition function for weighted context-free grammars has a convenient diagrammatic representation. For individual grammars, the behavior of a large text $ℓ ≫ 1$ is governed by saddle-point equations, which resemble belief-propagation equations [20].

We then considered two ensembles of grammars, which are equivalent in a large temperature limit. The Gaussian model (GM) was solved exactly, and shown to become unphysical for ${~ ϵ}_{d} log N ≲ 1 / 6$ . For the random language model (RLM), previously simulated in [1], the partition function was computed in the replica-symmetric ansatz; the entropy becomes negative at low temperature, again depending essentially on the quantity ${~ ϵ}_{d} log N$ . Finally, the order parameter $Q_{2}$ was computed in the replica-symmetric ansatz. The prediction quantitatively agrees with simulations above ${~ ϵ}_{d} log N ≳ 1$ . These results indicate that replica-symmetry must be broken in the nontrivial low-temperature phase.

The RLM bears some similarity to a spin-glass on the Bethe lattice, a difficult problem that is still not fully understood [21, 10, 22, 17, 23]. Indeed, both problems can be generated by a diagrammatic method, and in both problems one finds that overlaps of all orders are needed to compute the partition function. However, for the spin-glass, one can perform an expansion around the mean-field limit, which is the Sherrington-Kirkpatrick model solved by Parisi [19]. Naïvely, the analogue to the SK model would be the Gaussian model, which we solved above. However, we showed that this model does not break the replica symmetry. This is related to a gauge symmetry in the diagrammatic formulation. It is therefore an open question whether there is a more primitive model that captures the essence of random languages in the low-temperature phase, and remains solvable.

Finally, we have focussed here on context-free grammars, for which derivations are trees. The next level up in the Chomsky hierarchy are context-sensitive grammars. A theorem of Kuroda [24] says that it is sufficient to add rules of the form $a b \to c d$ to those above to model all context-sensitive grammars. Clearly this will add a quartic vertex to our (16), which is not in itself a difficulty. However, well-formed derivations must be represented by planar diagrams, so that the order of symbols is preserved in the derivation. Generating random planar graphs that are not trees requires matrices as integration variables; this strongly suggests that general grammars require the full machinery of complex matrix models.

References

[1] DeGiuli E 2018 to appear in Phys. Rev. Lett. arXiv preprint arXiv:1809.01201
[2] Carnie A 2013 Syntax: A generative introduction (John Wiley & Sons, Ltd.)
[3] Chomsky N 2002 Syntactic structures (Berlin: Walter de Gruyter)
[4] Hopcroft J E, Motwani R and Ullman J D 2007 Introduction to automata theory, languages, and computation 3rd ed (Boston, Ma: Pearson)
[5] Searls D B 2002 Nature 420 211
[6] Di Francesco P, Ginsparg P and Zinn-Justin J 1995 Physics Reports 254 1–133
[7] Kazakov V 1986 Physics Letters A 119 140–144
[8] Boulatov D and Kazakov V 1987 Physics Letters B 186 379–384 0370–2693
[9] Bachas C, De Calan C and Petropoulos P 1994 Journal of Physics A: Mathematical and General 27 6121
[10] Baillie C, Janke W, Johnston D and Plecháč P 1995 Nuclear Physics B 450 730–752
[11] Bessis D, Itzykson C and Zuber J B 1980 Advances in Applied Mathematics 1 109–157
[12] t’Hooft G 1974 Nuclear Physics. B 72 461–473
[13] Brezin E, Itzykson C, Parisi G and Zuber J 1978 Commun. math. Phys 59 35–51
[14] Le Guillou J C and Zinn-Justin J 2012 Large-order behaviour of perturbation theory vol 7 (Elsevier)
[15] Di Francesco P 2006 2D quantum gravity, matrix models and graph combinatorics (Springer) pp 33–88
[16] Eynard B 2016 Counting surfaces (Springer)
[17] Parisi G and Tria F 2002 The European Physical Journal B-Condensed Matter and Complex Systems 30 533–541
[18] Flajolet P and Sedgewick R 2009 Analytic combinatorics (cambridge University press)
[19] Mézard M, Parisi G and Virasoro M 1987 Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications vol 9 (World Scientific Publishing Company)
[20] Mezard M and Montanari A X 2009 Information, physics, and computation (Oxford University Press)
[21] De Dominicis C and Goldschmidt Y 1989 Journal of Physics A: Mathematical and General 22 L775
[22] Mézard M and Parisi G 2001 The European Physical Journal B-Condensed Matter and Complex Systems 20 217–233
[23] Parisi G 2017 Journal of Statistical Physics 167 515–542 0022–4715
[24] Kuroda S Y 1964 Information and Control 7 207–223
[25] Mackey D S and Mackey N 2003 On the determinant of symplectic matrices (Manchester Centre for Computational Mathematics)

Appendix A Solution of Gaussian model

We introduce $Q_{L}^{i j} = \sum_{a} L_{a}^{i} L_{a}^{j}$ as a new variable with a corresponding momentum $Λ_{L}^{i j}$ , and similarly for $Q_{R}^{i j} = \sum_{a} R_{a}^{i} R_{a}^{j}$ and $P^{i j} = \sum_{a} L_{a}^{i} R_{a}^{j}$ , with conjugate momenta $Λ_{R}$ and $Λ_{P}$ , respectively. Let us write $x_{a}^{i} = R e [L_{a}^{i}], y_{a}^{i} = I m [L_{a}^{i}]$ . The variables ${x_{a}^{i}, y_{a}^{i}}$ are Gaussian, with a coupling matrix diagonal in colour. For each $a$ , the coupling matrix is a $2 n \times 2 n$ matrix acting on $(x_{a}^{1}, \dots x_{a}^{n}, y_{a}^{1}, \dots, y_{a}^{n})$ ,

Z = [\begin{matrix} ^δ - g {^Λ}_{L} & - g i {^Λ}_{L} - g i {^Λ}_{L} & ^δ + g {^Λ}_{L} \end{matrix}],

(46)

where $^δ$ is the $n \times n$ identity matrix. It is easily verified that $Z$ is complex symplectic: $J = Z^{t} \cdot J \cdot Z$ , where

J = [\begin{matrix} 0 & ^δ -^δ & 0 \end{matrix}]

(47)

This implies that $Z^{- 1} = J^{- 1} \cdot Z^{t} \cdot J$ and, less obviously, $| Z | = 1$ [25]. Hence after integrating out ${L_{a}^{i}}$ there is no nontrivial entropic term from $log | Z |$ , nor does there appear the inverse of $Λ_{L}$ , as would naively be expected. In fact, after integrating out ${L_{a}^{i}}$ and ${R_{a}^{i}}$ the action remains linear in $Λ_{L}$ , $Λ_{R}$ , and $Λ_{P}$ . Hence these can be immediately integrated out and we find that

Q_{L}^{i j} = N L_{*}^{i} L_{*}^{j}, Q_{R}^{i j} = N R_{*}^{i} R_{*}^{j}, P^{i j} = N L_{*}^{i} R_{*}^{j}

(48)

We find

Emergence of order in random languages

Authors

Random Language Model: a path to principled complexity

The replica-symmetric free energy for Ising spin glasses with orthogonally invariant couplings

On the coexistence of competing languages

Ancestor-to-Creole Transfer is Not a Walk in the Park

Zipf's law emerges asymptotically during phase transitions in communicative systems

Emergence of a finite-size-scaling function in the supervised learning of the Ising phase transition

1 Partition functions

2 Energy, Entropy, and Order Parameters

3 Diagrammatic formulation

4 Grammar ensembles

4.1 Gaussian model

4.2 Rlm

4.3 Order parameter

5 Conclusion

References

Appendix A Solution of Gaussian model

Emergence of order in random languages

Authors

Related Research

Random Language Model: a path to principled complexity

The replica-symmetric free energy for Ising spin glasses with orthogonally invariant couplings

On the coexistence of competing languages

Ancestor-to-Creole Transfer is Not a Walk in the Park

Zipf's law emerges asymptotically during phase transitions in communicative systems

Emergence of a finite-size-scaling function in the supervised learning of the Ising phase transition

This week in AI

1 Partition functions

2 Energy, Entropy, and Order Parameters

3 Diagrammatic formulation

4 Grammar ensembles

4.1 Gaussian model

4.2 Rlm

4.3 Order parameter

5 Conclusion

References

Appendix A Solution of Gaussian model