Hidden-variable theories allege that a state of a quantum system, even if it is pure and thus contains as much information as quantum mechanics permits, actually describes an ensemble of systems with distinct values of some hidden variables. Once the values of these variables are specified, the system becomes determinate or at least more determinate than quantum mechanics says. Thus the randomness in quantum predictions results, entirely or partially, from the randomness involved in selecting a member of the ensemble. No-go theorems assert that, under reasonable hypotheses, a hidden-variable interpretation cannot reproduce the predictions of quantum mechanics.
In this paper, we examine two species of such theorems, value and expectation no-go theorems. The value approach originated in the work of Bell [2, 3] and of Kochen and Specker  in the 1960’s. Value no-go theorems establish that, under suitable hypotheses, hidden-variable theories cannot reproduce the predictions of quantum mechanics of the possible values of observables. The expectation approach was developed by Spekkens  and by Ferrie, Emerson, and Morris [9, 10, 11], with  giving the sharpest result. Expectation no-go theorems establish that, under suitable hypotheses, hidden-variable theories cannot reproduce the predictions of quantum mechanics of the expectations of observables.
The bold philosophical paper  of Spekkens attracted our attention. We started [6, 7] with repairing various flaws in the four papers on expectation no-go theorems. In [10, §VI.B], Ferrie and Emerson write that “Spekkens’ notion of non-contextuality is a generalization of the traditional notion initiated by Kochen and Specker.” This and other remarks strongly suggest that the expectation approach is more general than the value approach. We had our doubts about that.
In both the value and the expectation approach, measurements are associated to Hermitian operators, but they are different sorts of measurements. In the value approach, Hermitian operators serve as observables, and measuring one of them produces a number in its spectrum. In the expectation approach, certain Hermitian operators serve as effects. Here an effect is a positive operator dominated by the identity operator (so that is positive as well). Measuring an effect produces 0 or 1, even if the spectrum of consists entirely of other numbers. The only Hermitian operators for which these two uses coincide are projections.
We sharpen the results of both approaches so that only projection measurements are used. Regarding the expectation approach, we substantially weaken the hypotheses. Arbitrary effects are replaced with rank-1 projections. Accordingly, we need convex-linearity only for the hidden-variable picture of states, not for that of effects. Regarding the value approach, it turns out that rank-1 projections are sufficient in the finite dimensional case but not in general. Finally, using a successful hidden-variable theory of John Bell for a single qubit, we demonstrate that the expectation approach does not subsume the value approach.
A more comprehensive version of this paper, complete with the proofs, is found at .
2 Expectation No-Go Theorem
An expectation representation for quantum systems described by a Hilbert space is a triple where
is a measurable space,
is a convex-linear map assigning to each density operator on
a probability measureon , and
is a map assigning to each rank-1 projection in a measurable function from to the real interval .
It is required that for all density matrices and all rank-1 projections
The convex linearity of means that whenever are nonnegative real numbers with sum 1.
The definition of expectation representation is similar to Ferrie-Morris-Emerson’s definition of the probability representation  except that (i) the domain of contains only rank-1 projections, rather than arbitrary effects, and (ii) we do not (and cannot) require that be convex-linear.
Intuitively an expectation representation attempts to predict the expectation value of any rank-1 projection in a given mixed state . The hidden variables are combined into one variable ranging over . Further, is the probability measure on determined by , and is the probability of determining the effect at the subensemble of determined by . The left side of (1) is the expectation of in state predicted by quantum mechanics and the right side is the expectation of in the ensemble described by .
But why is supposed to be convex linear? Well, mixed states have physical meaning and so it is desirable that be defined on mixed states as well. If you are a hidden-variable theorist, it is most natural for you to think of a mixed state as a classical probabilistic combination of the component states. This leads you to the convex linearity of . For example, if where ’s are nonnegative reals and
then, by the rules of probability theory,for any measurable
. Note, however, that you cannot start with any wild probability distributionon pure states and then extend it to mixed states by convex linearity. There is an important constraint on . The same mixed state may have different representations as a convex combination of pure states; all such representations must lead to the same probability measure .
Theorem 2 (First Bootstrapping Theorem).
Let be a closed subspace of a Hilbert space . From any expectation representation for quantum systems described by , one can directly construct such a representation for systems described by .
Theorem 3 (Expectation no-go theorem).
If the dimension of the Hilbert space is at least 2 then there is no expectation representation for quantum systems described by .
We cannot expect any sort of no-go result in lower dimensions, because quantum theory in Hilbert spaces of dimensions 0 and 1 is trivial and therefore classical. By the First Bootstrapping Theorem, it suffices to prove Theorem 3 just in the case . But we find Ferrie-Morris-Emerson’s proof that works directly for all dimensions  instructive, and in the full paper  we adjust it to prove Theorem 3. The adjustment involves adding some details and observing that a drastically reduced domain of suffices. The adjustment also involves making a little correction. Ferrie et al. quoted an erroneous result of Bugajski  which needs some additional hypotheses to become correct. Fortunately for Ferrie et al., those hypotheses hold in their situation.
Remark 4 (Symmetry or the lack of thereof).
In view of the idea of symmetry or even-handedness suggested by Spekkens , one might ask whether there is a dual version of Theorem 3, that is, a version that requires convex-linearity for effects but looks only at pure states and does not require any convex-linearity for states. The answer is no; with such requirements there is a trivial example of a successful hidden-variable theory, regardless of the dimension of the Hilbert space. The theory can be concisely described as taking the quantum state itself as the “hidden” variable. In more detail, let be the set of all pure states. Let assign to each operator the probability measure on concentrated at the point
that corresponds to the vector. Let assign to each effect the function on defined by . We have trivially arranged for this to give the correct expectation for any effect and any pure state . The formula for is clearly convex-linear (in fact, linear) as a function of . Of course, cannot be extended convex-linearly to mixed states, so that Theorem 3 does not apply.
3 Value No-Go Theorems
Value no-go theorems assert that hidden-variable theories cannot even produce the correct outcomes for individual measurements, let alone the correct probabilities or expectation values. Such theorems considerably predated the expectation no-go theorems considered in the preceding section. Value no-go theorems were first established by Bell [2, 3] and then by Kochen and Specker ; we shall also refer to the user-friendly exposition given by Mermin . To formulate value no-go theorems, one must specify what “correct outcomes for individual measurements” means.
Let be a Hilbert space, and let be a set of observables, i.e., self-adjoint operators on . A valuation for in is a function assigning to each observable a number in the spectrum of , in such a way that is in the joint spectrum of whenever are pairwise commuting.
The intention behind this definition is that, in a hidden-variable theory, a quantum state represents an ensemble of individual systems, each of which has definite values for observables. That is, each individual system has a valuation associated to it, describing what values would be obtained if we were to measure observable properties of the system. A believer in such a hidden-variable theory would expect a valuation for the set of all self-adjoint operators on , unless there were superselection rules rendering some such operators unobservable.
Before we proceed, we recall the notion of joint spectra [4, Section 6.5].
The joint spectrum of pairwise commuting, self-adjoint operators on a Hilbert space is a subset of . If are simultaneously diagonalizable then iff there is a non-zero vector with for . In general, iff for every there is a unit vector with for .
For any continuous function , we have
if and only if vanishes identically on .
The proposition is implicit in the statement, on page 155 of , that “most of Section 1, Subsection 4, about functions of a one operator,” can be repeated in the context of several commuting operators. We give a detailed proof of the proposition in [7, §4.1].
If then there is a finite set of rank 1 projections for which no valuation exists.
The proof of Theorem 8 can be derived from the work of Bell [3, Section 5], and we do that explicitly in [7, §4.3]. The construction given by Kochen and Specker  provides the desired more directly. The proof of Theorem 1 in  uses a Boolean algebra generated by a finite set of one-dimensional subspaces of , and it shows that the projections to those subspaces constitute an of the required sort. Mermin’s elegant exposition [14, Section IV] deals instead with squares of certain spin-components of a spin-1 particle, but these are projections to 2-dimensional subspaces of , and the complementary rank-1 projections serve as the desired .
Theorem 9 (Second Bootstrapping Theorem).
Suppose are finite-dimensional Hilbert spaces. Suppose further that is a finite set of rank-1 projections of for which no valuation exists. Then there is a finite set of rank-1 projections of for which no valuation exists.
Intuitively, such dimension bootstrapping results are to be expected. If hidden-variable theories could explain the behavior of quantum systems described by the larger Hilbert space, say , then they could also provide an explanation for systems described by the subspace . The latter systems are, after all, just a special case of the former, consisting of the pure states that happen to lie in or mixtures of such states. But often no-go theorems give much more information than just the impossibility of matching the predictions of quantum-mechanics with a hidden-variable theory. They establish that hidden-variable theories must fail in very specific ways. It is not so obvious that these specific sorts of failures, once established for a Hilbert space , necessarily also apply to its superspaces .
Theorem 10 (Value no-go theorem).
Suppose that the dimension of the Hilbert space is at least 3.
There is a finite set of projections for which no valuation exists.
If the dimension is finite then there is a finite set of rank 1 projections for which no valuation exists.
The desired finite sets of projections are constructed explicitly in the proof . The finiteness assumption in part (2) of the theorem cannot be omitted. If is infinite, then the set of all finite-rank projections admits a valuation, namely the constant zero function. This works because the definition of “valuation” imposes constraints on only finitely many observables at a time.
Let’s say that a projection on Hilbert space is a rank- projection modulo identity if either is of rank or else
splits into a tensor productsuch that is finite-dimensional and has the form where is of rank and is the identity operator on . The proof of Theorem 10 gives us the following corollary.
If the dimension of the Hilbert space is at least 3 then there is a finite set of rank-1 projections modulo identity for which no valuation exists.
4 One successful hidden-variable theory
By reducing both species of no-go theorems to projection measurement, where measurement as observable and measurement as effect coincide, we made it easier to see similarities and differences. No, the expectation no-go theorem does not imply the value no-go theorem. But the task of proving this claim formally, say for a given dimension , is rather thankless. You have to construct a counter-factual physical world where the expectation no-go theorem holds but the value no-go theorem fails. There is, however, one exceptional case, that of dimension 2. Theorem 3 assumes while Theorem 10 assumes . So what about dimension 2?
Bell developed, in  and , a hidden-variable theory for a two-dimensional Hilbert space . Here we summarize the improved version of Bell’s theory due to Mermin , we simplify part of Mermin’s argument, and we explain why the theory doesn’t contradict Theorem 3.
In the rest of this section, we work in the two-dimensional Hilbert space . Let be the set of value maps for all the observables on (so that
is an eigenvalue of an observable). In each pure state , the hidden variables should determine a particular member of .
A value representation for quantum systems described by is a pair where
is a probability space and
a function on the pure states such that every is a map from into .
Further, we require that, for any pure state and any observable , the expectation of the eigenvalue of agrees with the prediction of quantum theory:
. Notice that, if a random variable (in our case, the eigenvalue ofin ) takes only two values, then the expected value determines the probability distribution. A priori we should be speaking about commuting operators and joint spectra but things trivialize in the 2-dimensional case. Recall Proposition 7 and notice that, in the 2-dimensional Hilbert space, if operators commute, then one of them is a polynomial function of the other.
There exists a value representation for the quantum systems described by the two-dimensional Hilbert space .
A natural question arises why Bell’s theory doesn’t contradict Theorem 3? This is related to the convex linearity requirement. To obtain an expectation representation, we must extend the value representation, constructed in the proof of Theorem 13, to all density matrices in a convex linear way. But no such extension exists. In the full paper , we give a simple example illustrating what goes wrong if one attempts to extend the value representation in a convex linear way. Thus, Bell’s example of a hidden-variable theory for 2-dimensional does not fit the assumptions in any of the expectation no-go theorems. It does not, therefore, clash with the fact that those theorems, unlike the value no-go theorems, apply in the 2-dimensional case.
-  John S. Bell (1964): On the Einstein-Podolsky-Rosen paradox. Physics 1, pp. 195–200, doi:10.1103/PhysicsPhysiqueFizika.1.195.
-  John S. Bell (1966): On the problem of hidden variables in quantum mechanics. Reviews of Modern Physics 38, pp. 447–452, doi:10.1103/RevModPhys.38.447.
-  Michael S. Birman & Michael Z. Solomjak (1987): Spectral Theory of Self-Adjoint Operators in Hilbert Space. Reidel. Originally in Russian, Leningrad University Press, 1980.
-  Andreas Blass & Yuri Gurevich (July 2017): Common denominator for value and expectation no-go theorems. arXiv:1707.07368.
-  Andreas Blass & Yuri Gurevich (March 2015): Spekkens’s Symmetric No-Go Theorem. arXiv:1503.08084.
-  Andreas Blass & Yuri Gurevich (Sept. 2015): On Hidden Variables: Value and Expectation No-Go Theorems. arXiv:1509.06896.
-  Sławomir Bugajski (1993): Classical frames for a quantum theory—a bird’s-eye view. International Journal of Theoretical Physics 32, pp. 969–977, doi:10.1007/BF01215303.
-  Christopher Ferrie & Joseph Emerson (2008): Frame representations of quantum mechanics and the necessity of negativity in quasi-probability representations. Journal of Physics A: Mathematical and Theoretical 41:352001, doi:10.1088/1751-8113/41/35/352001. See also arXiv:0711.2658.
-  Christopher Ferrie & Joseph Emerson (2009): Framed Hilbert space: hanging the quasi-probability pictures of quantum theory. New Journal of Physics 11:063040, doi:10.1088/1367-2630/11/6/063040. See also arXiv:0903.4843.
-  Christopher Ferrie, Ryan Morris & Joseph Emerson (2010): Necessity of negativity in quantum theory. Physical Review A 82:044103, doi:10.1103/PhysRev.40.749. See also arXiv:0910.3198.
-  Teiko Heinosaari (2013): A simple sufficient condition for the coexistence of quantum effects. Journal of Physics A: Mathematical and Theoretical 46:152002, doi:10.1088/1751-8113/46/15/152002.
-  Simon Kochen & Ernst Specker (1967): The problem of hidden variables in quantum mechanics. Journal of Mathematics and Mechanics 17, pp. 59–87.
-  N. David Mermin (1993): Hidden variables and the two theorems of John Bell. Reviews of Modern Physics 65, pp. 803–815, doi:10.1103/RevModPhys.65.803.
-  Robert Schatten (1970): Norm ideals of completely continuous operators. Springer Verlag, doi:10.1007/978-3-662-35155-0. 2nd edition.
-  Robert W. Spekkens (2008): Negativity and contextuality are equivalent notions of nonclassicality. Physics Review Lettters 101(2):020401, doi:10.1103/PhysRevLett.101.020401. See also arXiv:0710.5549.