Kolmogorov-Sinai entropy and dissipation in driven classical Hamiltonian systems

05/13/2018 ∙ by Matheus Capela, et al. ∙ UPV/EHU 0

A central concept in the connection between physics and information theory is entropy, which represents the amount of information extracted from the system by the observer performing measurements in an experiment. Indeed, Jaynes' principle of maximum entropy allows to establish the connection between entropy in statistical mechanics and information entropy. In this sense, the dissipated energy in a classical Hamiltonian process, known as the thermodynamic entropy production, is connected to the relative entropy between the forward and backward probability densities. Recently, it was revealed that energetic inefficiency and model inefficiency, defined as the difference in mutual information that the system state shares with the future and past environmental variables, are equivalent concepts in Markovian processes. As a consequence, the question about a possible connection between model unpredictability and energetic inefficiency in the framework of classical physics emerges. Here, we address this question by connecting the concepts of random behavior of a classical Hamiltonian system, the Kolmogorov-Sinai entropy, with its energetic inefficiency, the dissipated work. This approach allows us to provide meaningful interpretations of information concepts in terms of thermodynamic quantities.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Dissipation, which occurs when systems are driven out-of-equilibrium, is a fundamental subject in physics, since it is related to reversibility of physical processes. Significant developments has been achieved in the description of such systems, specially concerning their energetics Jarzynski ; Jarzynski1 ; Evans ; Crooks . Since the development of classical information theory shannon and Jaynes’ principle of maximum entropy jaynes

—the equilibrium probability distribution maximizes information transfer in the measurement process—, several deep links between information theory and thermodynamics have been discovered

john .

Based on the observation that logical irreversibility implies thermodynamic irreversibility landauer , a relation between the energy cost of computation, which is a physical process, and the algorithmic complexity was derived zurek . The algorithmic complexity of a system is defined as the shortest algorithm, measured in bits, whose output is the actual physical configuration of the system. Thus, it is a natural measure of randomness, deeply linked with the information metric, i.e. distance between strings of bits.

An important relation between dissipation and informational entropy was obtained in Ref. kawai . Considering a driven Hamiltonian system, initially in equilibrium, it was proven that the dissipated work due to the driven process is proportional to the Kullback-Liebler divergence (relative entropy) between the forward and backward probability densities —backward process defined by the time-reverse driven protocol. This result directly leads to the second law of thermodynamics, since relative entropy is a positive semi-definite quantity.

Another recent work has revealed that energetic inefficiency, i.e. the dissipated energy, and model inefficiency (non-predictive power) are equivalent concepts in Markovian processes still . There, predictive power is the information that the system retains from the past and that is actually necessary to predict the system future behavior jeffreys ; still1 ; still2 . Therefore, non-predictive information is just the difference between all the information that the systems has and the predictive power.

Using the mathematical formulation provided by Shannon shannon , Kolmogorov constructed a theoretical tool, currently known as Kolmogorov-Sinai entropy (KSE), allowing to analyze the random behavior of dynamical systems. It is a parameter of the dynamical system which provides a criterion to define chaos, since positive KSE is a feature of chaotic behavior frigg . Despite such construction, the connection between thermodynamic entropy —or even the informational one— and KSE for general systems remains unclear latora ; bologna ; massimo .

This raises the question whether randomness and energetic inefficiency are connected in the framework of classical physics. The purpose of this work is to derive a mathematical relation between the concept of random behavior of a classical Hamiltonian system, measured by the KSE, and its energetic inefficiency, measured by dissipated energy, in accordance with the fluctuation relations. This connection is built by describing a dynamical system in terms of communication theory. is unambiguous, it paves the way for a novel route to investigate quantities like work and heat in the framework of quantum thermodynamics.

The manuscript is organized as follows: we start by defining the class of systems under consideration, followed by a description of an informational-theoretic approach for dynamical systems, in which the concept KSE of the dynamics is introduced. Then, in the main part of the article, we present our result, which establishes a lower bound on the dissipation in terms of the randomness generated by the dynamics. We close the paper with a proper discussion regarding the application of the result in classical systems.

Ii Physical setup

The system is described by a Hamiltonian , with the set of generalized coordinates and canonical conjugate momenta in phase space . The control parameter is varied in time following an externally controlled protocol , denoted the work protocol. We assume that the system is initially in thermal equilibrium with a reservoir at inverse temperature , thus the distribution function is given by


where is the partition function and . During the action of the work protocol, the system is isolated from the reservoir. This means that the energy exchange between the system and the external world is determined by , and we denote this energy as the work .

The dynamics in phase space is deterministic and governed by the Hamilton equations

We denote the Hamiltonian flow, i.e., the time evolution map that determines the trajectory associated with each initial condition , by . Therefore, our dynamical system is the triplet , a probability space, , equipped with a one-parameter group of automorphisms of the probability measure space, , with each time evolution function depending on time  arnold ; sinai ; frigg . Additionally, is the initial probability measure over the sigma-algebra . For Hamiltonian systems, is simply the set of all the subsets of the phase space . Finally, we would like to highlight that, due to the Hamiltonian dynamics, Liouville’s theorem applies.

Let us now define, for every time , the functional


which is the Shannon differential entropy associated to the density at time cover . We observe that is defined considering the support of the probability distribution. Initially, as canonical equilibrium is assumed, this is equal to the thermodynamic entropy and thus, contains all the thermodynamic information about the system. However, in general, this is no longer true for , since the work protocol acting on the system may lead it out of equilibrium.

Iii Randomness

The main goal of the present study is to establish a connection between the dissipated energy during the work protocol and the Kolmogorov-Sinai entropy (KSE). Therefore, as we must define this entropy for the dynamical system, we should seek an information-based description of the dynamical system. Usually, in a communication process, the source emits to the receiver discrete symbols (the messages) according to certain probability distribution. The KSE quantifies the randomness of this process. The goal is to compute it for a dynamical system and associate randomness to chaos. Consequently, in order to relate KSE to a dynamical system, we need to define a discrete alphabet, which is accomplished by partitioning the phase space frigg .

We say that a collection of subsets is a partition of the phase space , if its elements are disjoint, i.e. , if , and cover the , . Let us remark that, from two given partitions, and , it is possible to define a new partition, , by means of the refinement .

Let us now focus on the Hamiltonian flow. In order to simplify the analysis, we will consider a discrete version of the dynamical system. This means that the time is discrete, , and the time evolution is generated by the iteration of the automorphism . Let us remark that such assumption is physically sensible and reasonable, since it is in accordance with the statement that any measurement is discrete in time from an experimental point of view. In this context, time evolution is introduced by means of the refinement of partitions as follows. Let us consider some initial partition , whose elements are denoted by . Then, an evolved partition is obtained by the application of the evolution map on , i.e. . Therefore, trajectories in phase space provide us with a discrete sequence of partitions , based on an initial partition . This is the required discrete alphabet to extend the definition of KSE for dynamical systems frigg .

From these definitions, the entropy of a finite partition with respect to the probability measure

can be defined as the Shannon entropy of the probability vector


where if , and if . We can interpret this entropy as the amount of uncertainty concerning the element of the partition in which the state is.

We are now ready to define the randomness of the dynamical system. Let us define the KSE associated with the dynamics as arnold


where and is the set of all finite partitions of the phase space. For the sake of clarity, we have denoted as the application of the inverse map . Indeed, note that the Hamiltonian flux is a bijection and hence, there exists its inverse. Consequently, we can compute all the transition probabilities only relying on the initial probability . If represents the probability measure at time , then . This implies that and Eq. (4) follows, where is the Shannon entropy computed at time .

Starting from some initial condition , we can follow the associated orbit and, at every instant of time , assigning an element of the partition, , which contains the state . In other words, for every initial condition and every partition, we can construct the path . Note that is an element of the original partition in which the system is at time , it does not necessarily mean that and are contiguous. This is done in order to avoid the introduction of an extra index to label the elements of the partition.

Given the above definitions, we are interested in the probability of observing a specific path . According to the theory of dynamical systems, such probability is defined as frigg ; sinai


from which the conditional probabilities can be calculated as


The KSE is a standard tool in the analysis of chaotic behavior, in a sense that it quantifies the notion of randomness in the coarse-grained phase-space dynamics. We can understand this statement by looking at the equation


The RHS of the above equation is known as the generalized Shannon entropy of the dynamical system. It measures our knowledge about the dynamics of the system frigg . Indeed, the higher the uncertainty about the dynamics, the larger the randomness generated by it.

Iv A lower bound on dissipation

It is known that coarse-graining increases informational entropy, since we are effectively discarding information benatti ; ana . Here, we are interested in studying the thermodynamic entropy production rate during the dynamical evolution of the system. We start by defining the conditional coarse-grained density of states in the phase space as


where we have defined the indicator function if and otherwise, and the phase space volume .

We focus now on our main objective, which is to connect the randomness generated by the dynamics with the dissipated work (thermodynamic entropy production). We know that the entropy production is related to an entropic quantity by means of the fluctuation relation kawai . Therefore, we need to establish the connection between this entropic quantity and the KSE given by Eq. (7). In order to achieve this goal, we will compute the averaged Shannon entropy of the density of states in phase space, given by Eq. (8), , which is lower bonded by (see Appendix A for details)






We note that we defined and . The renaming terms () of and are defined as usual.

We note that the tilde over any quantity means that we are looking at the time-reversed trajectory.

In the present setup, the dissipated work is given by , with denoting the relative entropy between its arguments, while stands for phase space average. Now, by taking the time average of this equation and also of Eq. (9), we can relate the obtained results with the KSE through Eq. (7) (see Appendix A for details). The result is, hence,


In this equation, we define as the reference free energy at time and . Equation (12) is the main result of this work.

iv.1 Examples

Let us consider as a first example of dynamical system the rotation of the unit disk in the two dimensional phase space. The Hamiltonian function that generates such dynamics is the quadratic form , where the dimensionless energy is constrained to be smaller than one. This is just a harmonic oscillator. Setting the unit of time as the time spent in a rotation of radians and considering the trivial partition , the quantity is equal to zero, since for integrable dynamics. The bound is then saturated in the infinite temperature limit. If we consider a different partition , where () represents the states in the unit disk with positive coordinate (non-positive coordinate ), we have , thus implying in a negative, trivial bound. This simple example highlight the fact that the result is dependent on the chosen partition, which is expected since KSE is partition dependent, being maximal for the generating one. This is the case in which our result is not informative, since we do not have external driven and the dissipated work is zero.

Let us now move to a more interesting example by considering the case of a chaotic system, the paradigmatic kicked top. The system Hamiltonian can be written in the form , where is -th component of the angular momentum and we choose a unit period of the kicks. It is easy to see that the total angular momentum is conserved, thus restricting the phase space to a sphere. Choosing the energy scale appropriately we have that the system is chaotic for , in the sense that the greatest Lyapunov exponent becomes positive. We are interested here in the complete chaotic regime, in which the entire phase space is chaotic. This happens for .

In this regime we can compute the time averages that appears in Eq. (12) by considering the fact that the system quickly thermalize, i.e. all points in phase space have the same probability. Let us start with , which is the most difficult quantity to numerically compute. In our case the trajectories of the system would be so sparse that, for any sequence , the system will assume equal probabilities for the event . Mathematically , where is the number of elements in the considered partition. Due to the same reason, the conditional volume reduces simply to . Plugging this into Eq. (IV) results in for the complete chaotic case.

Regarding the calculation of the quantity , consider the following reasoning, which is similar to the one used above for . First, let us point out that is just the probability of finding the system state in at time , as implied by Liouville’s theorem. Since the dynamics is chaotic, there is an instant of time that the probability is approximately equal to the volume . Then, the quantity is approximately equal to . If is a partition such that its cells have equal volume the quantity reduces to . Note that if tends to infinity, then tends to zero. Therefore, Eq. (12) reduces to


We can compute the value of KSE by means of Pesin identity. The for the case (and ) is .

Regarding the saturation of our result, we note that the inequality appearing in Eq. (12) of the main text is a consequence of the approximation considered in Eq. (14) of Appendix A. Therefore, inequality (12) saturates when the time average of coarse-grained entropy (averaged over all trajectories) is equal to the entropy itself. This implies that the KS entropy must be equal to the initial thermodynamic entropy (Boltzmann constant is equal to 1) per unit time.

V Discussion

We can understand Eq. (12) in the following way. The difference between the Shannon entropies before and after imposing the coarse-graining is called hidden information ana . This is the information ignored by coarse-graining the system. Hence, according to this definition, the quantity can be interpreted as the minimum amount of hidden information, since (). The equality holds when

is the uniform distribution. In this way, the quantity

represents the maximum information that is not hidden, i.e. the information which is not related to the coarse-graining procedure. This is the maximum value which the differential entropy can assume.

Following this line of reasoning, the quantity is naturally interpreted as the average hidden information. Thus, the quantity represents the minimum average hidden information, since . Therefore, the quantity represents the maximum average of the information that is not hidden information — is the partition that maximizes . This is the information generated by the system dynamics.

The study of out-of-equilibrium systems is in general a very hard task. For instance, the result in Ref. kawai established a powerful connections between information theory and thermodynamics, relating the dissipated work in a Hamiltonian process to the asymmetry between the time evolution of the system and its time-reverse counterpart. The purpose of the present study is to provide new tools for investigating the thermodynamics of out-of-equilibrium Hamiltonian systems. Indeed, based on the understanding of dynamical systems in terms of communication theory, we were able to derive a lower bound on the dissipated work (entropy production) in terms of the complexity of the dynamics, measured by the KSE. In other words, we build a connection between a dynamical quantity, KSE, and a macroscopic physical one, the dissipated work.

In the development of the present work we have assumed that Liouville’s theorem applies, which is the case for closed Hamiltonian systems. Based on some results in literature dealing with fluctuation relations for open classical systems Campisi , we do believe that our results can be extended to such scenario, including non-Markovian dynamics. However, the connection between communication theory and dynamical maps in this case must be defined very carefully.

Regarding the extension of our work to quantum systems, the author of Ref. lindblad defined a dynamical entropy by proposing an adequate definition of quantum stochastic processes. However, we have currently a much deeper understanding of such processes kavan , and establishing a connection between them and quantum communication theory should be possible. This path might lead us to an unambiguous definition of KS entropy in the quantum setting.

Acknowledgments — Discussions with Iñigo L. Egusquiza and Kavan Modi are greatly acknowledged. MC and LCC acknowledge financial support from the Brazilian funding agencies CNPq (Grants No. 401230/2014-7, 305086/2013-8 and 445516/2014-), CAPES (Grant No. 6531/2014-08), and the Brazilian National Institute of Science and Technology of Quantum Information (INCT/IQ). M.S. and E.S. are grateful for the funding of Spanish MINECO/FEDER FIS2015-69983-P and Basque Government IT986-16. This material is also based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advance Scientific Computing Research (ASCR), under field work proposal number ERKJ335. LC acknowledges the warm hospitality of QUTIS group at the University of the Basque Country.

Appendix A Derivation of the Lower Bound for the Dissipated Work

The Shannon entropy of averaged over all possible paths can be compared to the Shannon entropy of


where the inequality follows from the relation , . It is possible to rewrite the quantity considering the volume measure as a time-reversed measure. Since preserves volume measure due to Liouville’s theorem, it follows that


which will below lead to Eq. (IV) for . As the time rate of is equivalent to KSE, when calculated for the finite partition which maximizes it, it follows the lower bound on KSE






The free energy is defined as the Legendre transformation . Note that the initial distribution is a equilibrium one and thus, by using Liouville’s theorem, it follows the relation . Additionally, the dissipated work up to time , kawai , which can be calculated in processes like the ones considered here as


Finally, by solving for and using inequality (16), we achieve the desired lower bound given by Eq. (12).


  • (1) C. Jarzynski, Equalities and inequalities: Irreversibility and the second law of thermodynamics at the nanoscale, Ann. Rev. Condens. Matter 2, 329 (2017).
  • (2) C. Jarzynski, Nonequilibrium equality for free energy differences, Phys. Rev. Lett. 78, 2690 (1997).
  • (3) D. J. Evans, E. G. D. Cohen and G. P. Morriss, Probability of second law violations in shearing steady states, Phys. Rev. Lett. 71, 2401 (1993).
  • (4) G. E. Crooks, Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences, Phys. Rev. E 60, 2721 (1999).
  • (5) C. E. Shannon, A mathematical theory of communication, Bell System Technical Journal 27, 379 (1948).
  • (6) J. Goold, M. Huber, A. Riera, L. del Rio and P. Skrzypczyk, The role of quantum information in thermodynamics —a topical review, J. Phys. A: Math. Theor. 49, 143001 (2016).
  • (7) T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley & Sons, New Jersey, 2006).
  • (8) E. T. Jaynes, Information theory and statistical mechanics, Phys. Rev. 106, 620 (1957).
  • (9) S. Still, D. A. Sivak, A. J. Bell and G. E. Crooks, Thermodynamics of prediction, Phys. Rev. Lett. 109, 120604 (2012).
  • (10) R. Landauer, Irreversibility and heat generation in the computing process, IBM J. Res. Dev. 3, 113 (1961).
  • (11) W. H. Zurek, Thermodynamic cost of computation, algorithmic complexity and the information metric, Nature 341, 119 (1989).
  • (12) R. Kawai, J. M. R Parrondo and C. Van den Broeck, Dissipation: The phase-space perspective, Phys. Rev. Lett. 98, 080602 (2007).
  • (13) V. I. Arnold and A. Avez, Ergodic Problems of Classical Mechanics (WA Benjamin, 1968).
  • (14) R. Frigg, In what sense is the Kolmogorov-Sinai entropy a measure for chaotic behaviour? Bridging the gap between dynamical systems theory and communication theory, Brit. J. Phil. Sci. 55, 411 (2004).
  • (15) R. C. Tolman, The Principles of Statistical Mechanics (Oxford University Press, 1938).
  • (16) I. P. Cornfeld, S. V. Fomin and Ya. G. Sinai, Ergodic Theory (Springer-Verlag, 1982).
  • (17) J.-P. Eckmann and D. Ruelle, Ergodic theory of chaos and strange attractors, Rev. Mod. Phys. 57, 617 (1985).
  • (18) F. Benatti, Deterministic Chaos in Infinite Quantum Systems (Springer, 1960).
  • (19) A. Alonso-Serrano and M. Visser, Coarse graining Shannon and von Neumann entropies, Entropy 19, 207 (2017).
  • (20) H. Jeffreys, Theory of Probability (Oxford University, New York, 1998).
  • (21) S. Still, Information-theoretic approach to interactive learning, Europhys. Lett. 85, 28005 (2009).
  • (22) S. Still, J. P. Crutchfield and C. Ellison,

    Optimal causal inference: Estimating stored information and approximating causal architecture

    , Chaos 20, 037111 (2010).
  • (23) V. Latora and M. Baranger, Kolmogorov-Sinai entropy rate versus physical entropy, Phys. Rev. Lett. 82, 520 (1999).
  • (24) M. Bologna, P. Grigolini, M. Karagiorgis and A. Rosa, Trajectory versus probability density entropy, Phys. Rev. E 64, 016223 (2001).
  • (25) M. Falcioni, L. Palatella and A. Vulpiani, Production rate of the coarse-grained Gibbs entropy and the Kolmogorov-Sinai entropy: A real connection?, Phys. Rev. E 71, 016118 (2005).
  • (26) C. Jarzynski, H. T. Quan and S. Rahav, Quantum-classical correspondence principle for work distributions, Phys. Rev. X 5, 031038 (2015).
  • (27) M. Campisi, P. Hänggi, and P. Talkner. Colloquium: Quantum fluctuation relations: Foundations and applications. Rev. Mod. Phys. 83, 771 (2011).
  • (28) G. Lindblad, Non-Markovian quantum stochastic processes and their entropy, Comm. Math. Phys. 65, 281 (1979).
  • (29) F. A. Pollock, C. Rodríguez-Rosario, T. Frauenheim, M. Paternostro, and K. Modi. Non-Markovian quantum processes: Complete framework and efficient characterization. Phys. Rev. A 97, 012127 (2018).