1 Introduction
Most scientific papers start with an abstract that focuses on the main ideas of the work but leaves out many of the details. From an information theoretic point of view this allows for very efficient processing which is crucial if information processing capabilities are limited. In general, abstractions are formed by reducing the information content of an entity until it contains only information that is relevant for a particular purpose. This partial neglect of information can lead to different entities being treated as equal or, phrased differently, the separation of structure from noise. Consider the abstract concept of a “chair”, where many aspects such as the size, color, material or particular shape are considered as noise that is irrelevant to the purpose of “sitting down”.
The ability to form abstractions is thought of as a hallmark of intelligence, both in cognitive tasks and in basic sensorimotor behaviors [1, 2, 3, 4, 5, 6] Traditionally it is conceptualized as being computationally costly because particular entities have to be grouped together by neglecting irrelevant information. Here we argue that abstractions arise as a consequence of limited computational capacity. The inability to distinguish different entities leads to the formation of abstractions. Note that this information processing limitation can be induced through limited computational capacity, but also through limited sample sizes or low signaltonoise ratios. In this paper we study abstractions in the process of decisionmaking, where “similar” situations elicit the same behavior when partially ignoring the current situational context.
Following the work of [7]
decisionmaking with limited informationprocessing resources has been studied extensively in psychology, economics, political science, industrial organization, computer science and artificial intelligence research. In this paper we use a informationtheoretic model of decisionmaking under resource constraints
[8, 9, 10, 11, 12, 13, 14]. In particular, [15, 16, 17, 18] present a framework in which gain in expected utility is traded off against the adaptation cost of changing from an initial behavior to a posterior behavior. The variational problem that arises due to this tradeoff has the same mathematical form as the minimization of a free energy difference functional in thermodynamics. Here, we discuss the close connection between the thermodynamic decisionmaking framework [15] and ratedistortion theory which is an information theoretic framework for lossy compression. The problem in lossy compression is essentially the problem of separating structure from noise and is thus highly related to finding abstractions [19, 20, 21]. In the context of decisionmaking the ratedistortion framework can be applied by conceptualizing the decisionmaker as a channel from observations to actions with limited capacity, which is known in economics as the framework of “Rational Inattention” [22].In the next section we discuss how the ratedistortion framework can be obtained for boundedrational decisionmakers that face a number of tasks. In Section 3 we demonstrate two simple applications to explore the type of abstractions that emerge from limited information processing capabilities. In Section 4 we summarize the findings and discuss the presented approach.
2 Ratedistortion theory for decisionmaking
2.1 Boundedrational decisionmaking
In [15], a boundedrational actor that initially follows a policy changes its behavior to in a way that optimally trades off the expected gain in utility against the transformation costs for adapting from to . This tradeoff is formalized by the following variational principle
(1) 
where is known as the inverse temperature and is known as the difference in free energy—negative free energy in physics—which is composed of the expected utility w.r.t. and the KullbackLeibler (KL) divergence between and . acts as a conversionfactor between transformation cost (usually in nats or bits) and the expected utility.
The distribution that maximizes the variational principle is given by
(2) 
with the partition sum .
The influence of the transformation cost and thus the boundedness of the actor is governed by the parameter which determines “how far” the final behavior can deviate from the initial behavior measured in terms of KLdivergence. The perfectly rational actor that maximizes his utility can be recovered as the limit case where transformation cost is ignored, whereas corresponds to an actor that has infinite transformation cost or no computational resources and, thus, sticks with his prior policy .
Note that in the notation shown here, is conceptualized as a function over gains. In case
corresponds to a lossfunction, the same variational principle allows to find the distribution
that optimally trades off minimum expected loss against transformation cost. In this case the argmin over has to be taken and the sign of is inverted. In caseis a continuous random variable, sums have to be replaced by the corresponding integrals.
2.2 Multitask decisionmaking with limited resources
Consider an actor that is embedded into an environment and receives (potentially partial and noisy) information about the current state of the environment, that is the actor observes the value of a random variable . This observation allows the actor to reduce uncertainty about the current state of the environment and adapt its behavior correspondingly. Formally this is expressed with the conditional distribution over the action . The thermodynamic framework for decisionmaking introduced in the previous section can straightforwardly be harnessed for describing such a boundedrational agent that receives information by plugging in the conditional distribution into Equation 1
(3) 
with the solution
(4) 
Notice that the utility function in general depends on the observation , leading to , but to indicate the conditioning on a specific value of we write .
The initial distribution can be interpreted as a default or priorbehavior in the absence of an observation, thus we will refer to as “the prior”. The information processing cost is then given as the KL divergence between and the prior with the conversion factor that relates the units of transformation cost and the units of utility.
2.3 The optimal prior
In the free energy principle (Equation 3), the prior is assumed to be given. A very interesting question is which prior distribution maximizes the free energy difference for all observations on average. To formalize this question, we extend the variational principle in Equation 3 by taking the expectation over and the argmax over
The inner argmaxoperator over and the expectation over can be swapped because the variation is not over . With the KLterm expanded this leads to
The solution to the argmax over is given by . (see 2.1.1 in [19] or [23]). Plugging in for yields the following variational principle for boundedrational decisionmaking with a minimum average relative entropy prior
(5) 
where is the mutual information between and . The variational problem can be interpreted as maximizing expected utility with an upper bound on the mutual information or in the dual point of view, as minimizing the mutual information between actions and observations with a lower bound on the expected utility. The problem in Equation 5 is equivalent to the problem formulation in ratedistortion theory ([19, 24, 25]), where is usually conceptualized as a distortion function which leads to a flip in the sign of and an argmin instead of an argmax.
The solution that extremizes the variational problem is given by the selfconsistent equations (see [19])
(6)  
(7) 
Note that the solution for the conditional distribution in the ratedistortion problem (Equation 6) is the same as the solution in the free energy case of the previous section (Equation 4), except that the prior is now defined as the marginal distribution (see Equation 7). This particular prior distribution minimizes the the average relative entropy between and which is the mutual information between actions and observations .
In the limitcase where transformation costs are ignored, is equal to the perfectly rational policy for each value of independent of any of the other policies and
becomes a mixture of these solutions. Note that if there is a subset of perfectly rational solutions that is shared among tasks, then only this subset will be assigned probability mass since it reduces mutual information (see Section 3.3) Importantly, high values of the mutual information term in Equation
5 will not lead to a penalization, which means that actions can be very informative about the observation . The behavior of an actor with infinite computational resources will thus be very observationspecific.In the case the mutual information between actions and observations is minimized to , leading to , the maximal abstraction where all elicit the same response. The actor’s behavior becomes independent of the observation due to the lack in computational resources to change its behavior. Within this limitation the actor will, however, still emit actions that maximize the expected utility .
For values of the rationality parameter in between these limitcases, that is , the boundedrational actor trades off observationspecific actions that lead to a higher expected utility for particular observations at the cost of high mutual information between observations and actions , against abstract actions that yield a “good” expected utility for many observations and lead to a lower mutual information term.
An alternative interpretation, closer to the ratedistortion framework, is that the perceptual channel through which is transmitted to the actor has a limited capacity given by . For large values of , the transmission of is not severely influenced and the actor can choose the best action for this particular observation. For lower values of however, the actor becomes very uncertain about the true value of and has to choose abstract actions that are “good” under all observations which are compatible with the actor’s belief over .
2.4 Computing the selfconsistent solution
The selfconsistent solutions that maximize the variational principle in Equation 5 can be computed by starting with an initial distribution and then iterating Equation 6 and Equation 7 in an alternating fashion. This procedure is well known in the ratedistortion framework as a BlahutArimototype algorithm [25, 26]. The iteration is guaranteed to converge to a unique maximum (see 2.1.1 in [19] and [23, 24]. Note that has to have the same support as .
Implemented in a straightforward manner, the BlahutArimoto iterations can become computationally costly since the iterations involve evaluating the utility function for every actionobservationpair and computing the normalization constant . In case of continuousvalued random variables, closedform analytic solutions exist only for special cases.
3 Abstractions in multitask decisionmaking
3.1 Problem formulation
In the following we present the application of the ratedistortion framework for decisionmaking introduced in the previous section to multitask decision problems. We assume that we are given a number of tasks within the same environment and that the observations from the environment are fully informative about the current task, that is we observe the value of a discrete random variable
corresponding to a unique task. Note that this assumption can easily be relaxed.More formally we make the following assumptions: we are given a set of tasks which define the set of observations with if and only if . Each task is defined through the utility function , where is an action. The actionspace is the same for all tasks. We assume that the probability over tasks is known and given by .
The goal of the decision maker is to find taskspecific distributions that maximize the expected utility given its computational constraints. This problem is formalized in the variational principle in Equation 5 with the selfconsistent solutions in Equations 6, 7. In this principle for boundedrational decisionmaking, information processing costs arise from changing the priorbehavior to the taskspecific behavior and are measured in terms of KLcost in accordance with the thermodynamic framework for decisionmaking [15].
3.2 Trading off abstraction against optimal action
We designed the following twotask problem, to demonstrate the role of the rationality parameter that governs the tradeoff between expected utility and mutual information. In both tasks, the action
is one of four possible actionvectors (see Table
1). The utility for the first task is simply given by the value of the first component of the action vector, whereas the utility for the second task is the Manhattan distance between the two components of the action vector:The utilities for all actions are summarized in Table 1. The observationvariable is fully informative about the task with the task probabilities .
With this particular choice of utility functions and actionvectors, the maximumutility action for one task has a utility of zero for the other task. However, there is a suboptimal action that leads to the secondbest utility in both environments. The simulation results summarized in Table 1 show that for a high value of the inverse temperature the decisionmaker picks the maximumutility action in each task with probability . At a low value of the actor uses the same action distribution for both tasks due to its boundedness, resulting in . This leads to a maximal abstraction over both tasks which is solved optimally by putting all the probability mass on the suboptimal action . Note that the limit shown here is in general still far from the fully bounded limit — in this particular example however lowering further has no effect.
Figure 1 A shows the transition from perfect rationality to full boundedness. Starting at the entropy of the conditionals is zero, since for a given task the actor picks the maximumutility action with certainty. By lowering the inverse temperature , both the mutual information and the expected utility monotonically decrease. Initially stays constant, whereas increases, which means that the actor picks the two maximumutility actions with increasing stochasticity. At
a phase transition occurs — the entropy
rapidly peaks at bits implying that three actions are now equally probable in . Lowering further leads to a rapid drop in , and to zero bits as well as a drop in expected utility to . The decision maker is now in the fully abstract regime, where is always chosen, regardless of the task.Figure 1 B shows the RateUtility function (in analogy to the ratedistortion function) where the information processing rate is shown as a function of the expected utility. If the decisionmaker is conceptualized as a communication channel between observations and actions, the rate defines the minimal required capacity of that channel. The RateUtility function thus specifies the minimum required capacity for computing an action with a certain expected utility, or analogously the maximally achievable expected utility given a certain information processing capacity. Importantly, decisionmakers in the shaded region are impossible, whereas decisionmakers in the white region are suboptimal with respect to their information processing capabilities.
3.3 Changing the level of granularity
Abstractions are formed by reducing the information content of an entity until it only contains relevant information. For a discrete random variable this translates into forming a partitioning over the space where “similar” elements are grouped into the same subset of and become indistinguishable within the subset. In physics changing the granularity of a partitioning to a coarser level is known as coarsegraining which reduces the resolution of the space in a nonuniform manner. In the ratedistortion framework the partitioning emerges in the shared prior as a softpartitioning (see [20]), where actions with the same average utility get the same probability mass and become essentially indistinguishable.
To demonstrate this, we use a binary grid of size , where each cell of the grid can be white or colored in black . Actions are particular patterns on this grid thus the actionspace becomes . The utility function defines the following three tasks:

The utility equals the number of colored pixels, but one row and one column has to be allwhite, otherwise the utility is zero.

Any pattern with exactly four colored pixels scores a utility of , all other patterns have utility zero.

Any pattern with an even number of colored pixels scores a utility equal to the total number of colored pixels; all other patterns have a utility of zero.
Figure 2 shows samples each from the conditionals for each task and the prior for . Since the inverse temperature is high, all the samples with nonzero probability are actions that yield maximum utility in their particular task. Note that the patterns that lead to a maximum utility in task (1) are a subset of the patterns that lead to maximum utility in task (2) but also lead to a nonzero utility in task (3). Since transformation costs are mostly ignored in this case, the patterns appearing for task (3) are very different from the patterns in task (1). Note however that the additional patterns in task (2) that would also lead to maximum utility are assigned a probability of zero. The subset of patterns which are also optimal in task (1) is sufficient to achieve maximum expected utility and by not including the additional “specialized” patterns for task (2) the mutual information can be reduced significantly. The prior consists essentially of two kinds of patterns: the ones that are optimal in task (1) and (2) simultaneously and the patterns that are optimal in task (3). The first two tasks have essentially become indistinguishable because the actor will respond with exactly the same actiondistribution.
By lowering the inverse temperature to (see Figure 3), the mutual information constraint gets more weight and suboptimal patterns are picked for task (3), similar to the simulation in the previous section. The behavior of the actor has now become indistinguishable for all three tasks at the expense of a lower expected utility. Importantly, the effective resolution of the prior has reduced from two distinct sets of patterns to a single set of indistinguishable patterns (in terms of their expected utility). The level of granularity of the prior has been reduced even further.
4 Discussion & Conclusions
In this work, we discussed the connection between the thermodynamic framework [15] for decisionmaking with information processing costs and ratedistortion theory. This connection implies a novel interpretation of the ratedistortion framework for multitask boundedrational decisionmaking. Importantly, abstractions emerge naturally in this framework due to limited information processing capabilities. The authors in [27] find a very similar emergence of “natural abstractions” and “ritualized behavior” when studying goaldirected behavior in the MPD case using the Relevant Information method, which is a particular application of ratedistortion theory.
Although not shown here, the approach presented in this paper straightforwardly carries over to an inference case by treating as observations and as the beliefstate. In the inference case, limited information processing capacities make it impossible to detect certain patterns which in turn renders different entities indistinguishable, leading to the formation of abstractions. This idea has been explored previously in [20, 21]. Both, the work just mentioned and our work are inspired by the Information Bottleneck Method [19], which is mathematically very similar to the ratedistortion problem (with a particular choice of distortion function) and thus also to the approach presented here.
Note that limited information processing capabilities can arise for various reasons. The most obvious reason, perhaps, is the lack of computational power which is in many cases equivalent to certain timeconstraints (such as reaction times) or memory constraints. Other reasons for information processing limits are small sample sizes or low signaltonoise ratios that put an upper limit on the mutual information independent of available computational power.
In the approach presented here, we assume that the decisionmaker draws samples from . Responding to a certain task with a sample from could then be implemented for instance with a rejection sampling procedure. The prior will then be the proposaldistribution that has the highest average acceptance rate over all tasks . The computational cost of finding is not part of the current framework. These implications have to be explored in further work.
Acknowledgments
This study was supported by the DFG, Emmy Noether grant BR4164/11.
References
 [1] Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1279–1285, 2011.
 [2] Charles Kemp, Amy Perfors, and Joshua B Tenenbaum. Learning overhypotheses with hierarchical bayesian models. Developmental science, 10(3):307–321, 2007.
 [3] Samuel J Gershman and Yael Niv. Learning latent structure: carving nature at its joints. Current opinion in neurobiology, 20(2):251–256, 2010.
 [4] Daniel A Braun, Carsten Mehring, and Daniel M Wolpert. Structure learning in action. Behavioural brain research, 206(2):157–165, 2010.
 [5] Daniel A Braun, Stephan Waldert, Ad Aertsen, Daniel M Wolpert, and Carsten Mehring. Structure learning in a sensorimotor association task. PloS one, 5(1):e8973, 2010.
 [6] Tim Genewein and Daniel A Braun. A sensorimotor paradigm for bayesian model selection. Frontiers in human neuroscience, 6, 2012.
 [7] Herbert A Simon. Theories of bounded rationality. Decision and organization, 1:161–176, 1972.
 [8] Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games. Games and economic behavior, 10(1):6–38, 1995.

[9]
David H Wolpert.
Information theorythe bridge connecting bounded rational game theory and statistical physics.
In Complex Engineered Systems, pages 262–290. Springer, 2006.  [10] Hilbert J Kappen. Linear theory for control of nonlinear stochastic systems. Physical review letters, 95(20):200201, 2005.
 [11] Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In AAAI, 2010.
 [12] Emanuel Todorov. Efficient computation of optimal actions. Proceedings of the national academy of sciences, 106(28):11478–11483, 2009.

[13]
Evangelos Theodorou, Jonas Buchli, and Stefan Schaal.
A generalized path integral control approach to reinforcement learning.
The Journal of Machine Learning Research
, 9999:3137–3181, 2010.  [14] Jonathan Rubin, Ohad Shamir, and Naftali Tishby. Trading value and information in mdps. In Decision Making with Imperfect Decision Makers, pages 57–74. Springer, 2012.
 [15] Pedro A Ortega and Daniel A Braun. Thermodynamics as a theory of decisionmaking with informationprocessing costs. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Science, 469(2153), 2013.
 [16] Daniel A Braun, Pedro A Ortega, Evangelos Theodorou, and Stefan Schaal. Path integral control and bounded rationality. In Adaptive Dynamic Programming And Reinforcement Learning (ADPRL), 2011 IEEE Symposium on, pages 202–209. IEEE, 2011.
 [17] Daniel Alexander Ortega and Pedro Alejandro Braun. Information, utility and bounded rationality. In Artificial General Intelligence, pages 269–274. Springer, 2011.
 [18] Pedro A Ortega and Daniel A Braun. Free energy and the generalized optimality equations for sequential decision making. arXiv preprint arXiv:1205.3997, 2012.
 [19] Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. The 37th annual Allerton Conference on Communication, Control, and Computing, 1999.
 [20] Susanne Still and James P Crutchfield. Structure or noise? arXiv preprint arXiv:0708.0654, 2007.

[21]
Susanne Still, James P Crutchfield, and Christopher J Ellison.
Optimal causal inference: Estimating stored information and approximating causal architecture.
Chaos: An Interdisciplinary Journal of Nonlinear Science, 20(3):037111–037111, 2010.  [22] Christopher A Sims. Implications of rational inattention. Journal of monetary Economics, 50(3):665–690, 2003.
 [23] I Csiszár and G Tusnády. Information geometry and alternating minimization procedures. Statistics and decisions, 1984.
 [24] Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 1991.
 [25] Raymond W Yeung. Information theory and network coding. Springer, 2008.
 [26] Richard Blahut. Computation of channel capacity and ratedistortion functions. IEEE Transactions on Information Theory, 18(4):460–473, 1972.
 [27] Sander G van Dijk and Daniel Polani. Informational constraintsdriven organization in goaldirected behavior. Advances in Complex Systems, 2013.
Comments
There are no comments yet.