Statistical language modelling, whose aim is to capture the joint probability distribution of sequences of words, has applications to problems including information retrieval, speech recognition, artificial intelligence, human-machine interfaces, translation, and natural language problems that involve incomplete information. Early successes of statistical language models in industry include next-word prediction and vector embeddings of words based on colocation with reasonable performance on word similiarity exams. Efforts to build on early successes encounter difficulties arising from the high-dimensionality of the data—the number of meaningful texts in a lanugage is exponentially smaller than the number of texts that a room full of randomly typing monkeys could produce. One approach to address “the curse of high-dimensionality” is to truncate sequences under consideration to finite length phrases, or
-grams, and employ a hidden Markov model. Since hidden Markov models essentially cutoff correlations between words beyond a fixed distance, the success of this approach depends on the application. For example,-gram and -gram models have been employed effectively for speech recognition and translation, applications in which very long distance correlations are of limited importance . However, -gram models belie the essential critical behavior inherent in language 
. Human languages, like many biological systems including families of proteins, genomes, neurons in the brain, and the notes in a musical symphony, have significant long-range correlations that decay with a power law[4, 5, 6]. In contrast, any Markov or hidden-Markov system, such as an -gram model, has long range correlations that decay exponentially.
Recently, long-short term memory (LSTM) recurrent neural networks have been employed to produce statistical language model applications that considerately outperform those based on hidden Markov models. Notably, Google’s Neural Machine Translation System and the technology in Google Voice  have advanced the state of the art in both translation and speech recognition. While much is unknown about how these deep networks operate, new research indicates that it is hard to match long range and higher order statistics of natural languages even with LSTM recurrent neural networks . For certain applications such as translating a few phrases, matching higher order statistics may not be very important, but for other artificial intelligence applications, such as machine determination of humorous or deceptive narratives, they are essential.
In order to develop a statistical language model capable of capturing the higher order statistics of language, we turn to quantum statistical physics, which contains models such as solvable lattice models that exhibit correlation functions that decay with the power law—the same kind of critical behavior as biological systems. Unlike classical statistical physics, one spacial dimension suffices to exhibit criticality in quantum statistical physics . So, even a one dimensional quantum statistical language model could be a better alternative to LSTM recurrent neural networks, which may be likened to classical statistical physics. Entanglement in a quantum many body system is the metaphorical vehicle for statistical correlation in language, and serves as the proposed method to attack the high-dimensionality of the data. The number of basis states in a quantum many body system makes the state-space far too large to work with entirely but the number of physically relevant states occupy a subspace of exponentially smaller dimension, accessible by restricting to low-energy.
In this paper, we introduce a simple translation-invariant quantum statistical language model on a one-dimensional lattice which we call a trace-density model. While this trace-density model isn’t critical, it is an experimental first step toward a critical quantum model of language. The model involves matrix-product-states (MPS) which can approximate power–law decays “over quite long distances” 
. Two cubic constraintes are introduced. These constraints are physically motivated and mathematically related to the moment map of the action of the unitary groupon complex matrices. Training algorithms based on maximizing entropy and minimizing energy, similar to what’s described in [12, 13] can be developed and density matrix renormalization 
can be used. The model is introduced first using representation theoretic language and then interpretted using the physical language of MPS. A related language model based on an isometric tensor network is considered in.
The authors would like to thank Maxim Kontsevich and Miles Stoudenmire for helpful and stimulating discussions. The research of V.P. on this project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (QUASIFT grant agreement 677368); J.T. was supported in part by a grant from the US Army, Research, Development and Engineering Command, Mathematical Sciences Division, 711567-MA-II; and Y.V. received funding from Simons Foundation Award 385577.
2. The trace-density model for language
A mathematical model called a trace-density model for a corpus of text will be described in three steps. First, the input structures related to a corpus of text will be explained and some notation will be fixed. Second, the concept of a trace-density representation for a corpus will be defined. Third, the property that a representation must possess in order to be considered a trace-density model will be given.
2.1. The structures in a corpus of text
A corpus of text
is a finite sequence of words from a vocabulary . We refer to the elements of the vocabulary as words, and denote typical words with the letter , but the vocabulary can be comprised of any symbols (letters, bits, …) representing the atomic elements from which the corpus is constructed as a sequence. Let denote the set of all phrases contained in the corpus , a phrase being a finite subsequence consisting of adjacent elements. The set of phrases has more structure than merely a set. It is graded—each phrase has a well defined length given by the number of words comprising that phrase and the set of phrases is the disjoint union of sets phrases with same word length:
Also, has a (partially defined) product defined by concatanation of phrases. By adding a formal zero phrase to each graded component of , the product can be extended to all of by defining the product of two phrases to be zero if the concatanation is not a phrase contained in the corpus. The product is compatible with the grading
Viewed with the structure, the set of phrases is a graded monoid, a quotient, in fact, of the free monoid generated by the vocabulary .
Moreover, for each phrase length , there is a probability distribution, call it , on the set . Explicitly, is the number of times the phrase appears in the corpus , divided by , the total number of phrases of length . If the corpus is sufficiently large, this probability distribution is considered an approximation to a nonexistent, idealized probability distribution on all phrases in the language, of which the corpus is an observed sample. The goal is to model the collection of these idealized probability distributions.
2.2. Trace density representations
A density on a Hilbert space is a positive semi-definite operator (in the scope of this paper we do not assume that density is normalized by unit trace). Here, we fix a finite dimension and work entirely with with its standard inner product so that operators are identified with matrices. A density defines a nonnegative real valued function on the set of complex matrices by . Here, the superscript denotes complex conjugate transpose and denotes trace. The element of is the nonegative real number where is the -th row of the matrix , and is the sum of these numbers as ranges over the columns of .
A -dimensional trace density representation of a corpus consists of the following data:
A pair of densities , on such that
a function called the dictionary.
The dictionary assigns a complex matrix to each word in the vocabulary and extends to a function on all phrases by mapping a phrase to the product of matrices assigned to the words that comprise the phrase. That is, the phrase is mapped to the matrix . The trace density of the trace density representation is the nonegative, real valued function defined as the composition:
Notice that because for any two positive semi-definite operators it holds that 111Indeed, for positive semi-definite operators let and be Hermtian operators such that , , then , and is positive semi-definite operator.
2.3. The trace-density model of a corpus of text
A trace density representation for a corpus of text will be considered a trace density model for that corpus if the trace density of a phrase approximates the probability of the phrase appearing; that is, if for all phrases . The nature of the approximation and its dependence on is left vague.
A trace density model for a corpus captures the joint probability distribution of sequences of words efficiently since all of the information is contained in the dicitonary that translates individual words to matrices. The model requires no additional memory to store the matrices assigned to phrases, sentences, paragraphs, etc… Only a relatively efficient computation is required: a matrix product that is polynomial in the dimension of the representation and linear in the number of words.
Note also that a trace density model is translation invariant. The probability of the phrase appearing is independent of its position in the corpus.
2.4. Graphical language for tensor networks
Fix a corpus with vocabulary and let denote the complex -dimensional vector space generated by the vocabulary. Let . The dictionary of a -dimensional trace-density representation can be assembled into a single map by extending the assignment linearly. The single map is described by complex numbers and defines a tensor of order . A particular number and the entire tensor can be denoted graphically using a tensor network diagram as
These tensor network diagrams are oriented and rotating a labeled diagram upside down indicates complex conjugation. Connected edges denote contraction of indices. In this pictorial language, the following diagram represents the product of matrices
and the number is depicted
where the circular nodes depict the densities . The presence of the particular indices indicates that they are not summed over. The condition that is pictured as
3. Density and identity constraints
We now describe a pair of technically important constraints on a trace-density representation called the left density constraint and the right density constraint. These constraints guarantee that a trace-density has the abstract properties required of the joint-probability distributions on phrases in a language. These constraints also fit into a physical interpretation of our model which we describe in Section 4.
3.1. The right density constraint
Let be a corpus of text with vocabulary of words. Consider a -dimensional trace-density representation of with density and dictionary that maps . The trace-density representation satisfies the right density constraint provided
In tensor network notation, the density constraint is
The trace-density of a phrase is a nonnegative real number, but with no further a priori restrictions. The density constraint (1), however, implies that the trace-density of a representation gives rise to probability distributions on the set of length phrases .
If is the trace density of a representation satisfying the density constraint (1) then for every
Applying trace to the density constraint says the sum of trace densities over all vocabulary words is one:
Here is the corresponding picture
which, applied repeatedly, proves the theorem:
3.2. The left density constraint
Let be a corpus of text with vocabulary of words. Consider a -dimensional trace-density representation of with density and dictionary . The trace-density representation satisfies the left density constraint provided
The tensor network picture of the left density constraint is
Now, the right density constraint (1) together with the left density constraint (2) imply that the probability distributions defined by the trace-density fit together the way joint probabilities for sequences of words do; the probability distributions for phrases of different length are related as marginal probability distributions.
For a fixed phrase , the argument begins with the picture
and repeatedly use the left density constraint to reduce the left and the right density constraint to reduce the right yielding
Note that the left and right density constraints are cubic in the entries of the matrices . Yet, these two constraints imply the infinitely many higher order constraints stated in Proposition 1 and Proposition 2
, which are required for the joint distributions determined by the trace-density to fit together the way they must for statistical language model.
4. Quantum physical interpretation of trace density models
This section relates a quantum physical interpretation of a trace density model for language. Imagine a word as a quantum system consisting of a single particle having possible states—each word in the vocabulary being a possible state. Let be the -dimensional complex vector space generated by the vocabulary . The space becomes a Hilbert space with inner product defined by declaring that the vocabulary defines an orthonormal, independent spanning set of basis vectors.
The Hilbert space for a quantum many body system consisting of interacting particles is with an orthonormal basis consisting of the vectors . A state of such a many body system is a unit trace density and the probability that such a system is observed in the state is where is the projection on . A density induces a density on for by partial trace.
The hypothesis is that language is well described statistically by a pure state density in a tensor product of a very large number of copies of , which means that there exists such that is projection operator on .
A pure state in a tensor product of copies of is sometimes well approximated by a matrix product state (MPS). This means that there are auxillary spaces and vectors , , , …, , with obtained from by contracting all adjacent pairs in the expression
Note the decomposition of as an MPS is not unique. Even for fixed auxillary spaces, , there is a large gauge group acting the MPS decomposition. For example, automorphisms of each act nontrivially on the MPS decomposition, while fixing the state obtained after contraction.
A trace-density model for language attempts to approximate the pure state by a translation invariant MPS. Putting aside for the moment what happens at the far left and far right boundaries, a translation invariant MPS involves a single auxillary space of dimension (called the bond dimension) and a single tensor so that is obtain from by contracting in adjacent pairs. The space is isomorphic to , precisely the data of a dictionary. Training of translationally invariant MPS model is discussed in [15, 16].
Now, let us consider the boundary conditions. Assume is a state in an essentially infinite number of copies of . Then, for any finite , by partial trace, induces a state on . Tracing out the far left and far right yields a density as pictured below
Replacing the left and right boundaries by left and right densities, one obtains
Note that if is identity so that
then we can also interpret the collection as a collection of measurement operators on the Hilbert space with unit trace density , see  page 102.
5. Finding trace-density models
Trace-density representations satisfying both the right density and the left density constraints are plentiful. The simple trace-density representation consisting of
shows that representations satisfying both the left density and the right density constraints exist. To describe the moduli space of constrained representations, note that the left density constraint has the form of the isometry constraint on if the tensor is considered as a map with the standard Hermitian metric on and Hermitian metric on defined by . So, for a fixed the space of isometric tensors (i.e. those that satisfy the left density constraint) form homogeneous space
Moreover, there is automorphism group on preserving the Hermitian form. Modulo action of the automorphism group the moduli space of tensors that satisfy the left density constraint is Grassmanian of -dimensional complex planes in -dimensional complex space
Given a tensor satisfying the left identity constraint, an appropriate positive density that fits with the right density constraint can be found, as the following theorem proves.
Let be a tensor of order satisfying the left density constraint (2) with . Then there exists a right density so that together with satisfies the right density constraint.
Let be any tensor of order and consider the operator defined by
(1) For any , and so preserves self-adjoint matrices.
(2) For any , and so preserves positive semi-definite matrices.
(3) We have so if satisfies the left density constraint (2), the operator preserves the hyperlane in the space of matrices defined by the linear equation .
Combining (1), (2) and (3) we obtain that is a map of the set of positive semi-definite self-adjoint matrices that satisfy constraint to itself. Moreover, is continuous because it is a linear operator.
Since the set
is convex (a hyperplane section of a convex set of positive semi-definite operators), it is homeomorphic to a closed ball, then Brauer’s fixed point theorem implies that there exists a densitysuch that ∎
The left density and right density constraints make it possible to numerically find a trace-density model by a maximum log-likelihood algorithm . The idea is to find a trace-density representation that maximizes the (logarithm of the) trace-density for a training corpus. Intuitively, the constraints make certain that the total trace density over all possible phrases of a fixed length will equal one, and so maximizing the trace-densities of the phrases in the corpus will automatically make the exponential number of nonsense phrases have nearly zero trace-density.
-  J. T. Goodman, “A bit of progress in language modeling,” Computer Speech and Language 15 (2001), no. 4 403 – 434.
-  A. D. Brown and G. E. Hinton, “Products of hidden markov models.,” in AISTATS, Citeseer, 2001.
-  G. Zipf, Human behavior and the principle of least effort: an introduction to human ecology. Addison-Wesley Press, 1949.
-  R. Barbieri and M. Shimono, “Criticality in large-scale brain fmri dynamics unveiled by a novel point process analysis,” Networking of Psychophysics, Psychology and Neurophysiology (2012) 61.
-  T. Mora and W. Bialek, “Are biological systems poised at criticality?,” Journal of Statistical Physics 144 (2011), no. 2 268–302.
-  Ł. Dȩbowski, “Is Natural Language Strongly Nonergodic? A Stronger Theorem about Facts and Words,” ArXiv e-prints (June, 2017) 1706.04432.
-  Y. W. et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation,” CoRR abs/1609.08144 (2016).
-  H. Sak, A. W. Senior, and F. Beaufays, “Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition,” CoRR abs/1402.1128 (2014).
-  H. W. Lin and M. Tegmark, “Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language,” ArXiv e-prints (June, 2016) 1606.06737.
-  V. Korepin, “Universality of entropy scaling in one dimensional gapless models,” Phys. Rev. Lett. 92 (2004), no. 9.
E. Miles Stoudenmire and D. J. Schwab, “Supervised Learning with Quantum-Inspired Tensor Networks,”ArXiv e-prints (May, 2016) 1605.05775.
R. Bailly, “Quadratic weighted automata:spectral algorithm and likelihood
Proceedings of the Asian Conference on Machine Learning(C.-N. Hsu and W. S. Lee, eds.), vol. 20 of Proceedings of Machine Learning Research, (South Garden Hotels and Resorts, Taoyuan, Taiwain), pp. 147–163, PMLR, 14–15 Nov, 2011.
-  Z.-Y. Han, J. Wang, H. Fan, L. Wang, and P. Zhang, “Unsupervised Generative Modeling Using Matrix Product States,” ArXiv e-prints (Sept., 2017) 1709.01662.
-  V. Pestun and Y. Vlassopoulos, “Tensor network language model,” ArXiv e-prints (Oct., 2017) 1710.10248.
-  I. P. McCulloch, “Infinite size density matrix renormalization group, revisited,” ArXiv e-prints (Apr., 2008) 0804.2509.
-  G. M. Crosswhite, A. C. Doherty, and G. Vidal, “Applying matrix product operators to model systems with long-range interactions,” Physics Review B 78 (July, 2008) 035116, 0804.2504.
-  M. A. Nielsen and I. Chuang, Quantum computation and quantum information. AAPT, 2002.
-  R. Rosenfeld, “A maximum entropy approach to adaptive statistical language modelling,” Computer Speech and Language 10 (1996), no. 3 187 – 228.