Relaxing the assumption of independence is an active area of research in the statistics and machine learning literature. For time series, independence is replaced by the asymptotic independence of events far apart in time, or “mixing”. Mixing conditions make the dependence of the future on the past explicit, quantifying the decay in dependence as the future moves farther from the past. There are many definitions of mixing of varying strength with matching dependence coefficients (see[8, 6, 3] for reviews), but most of the results in the learning literature focus on -mixing or absolute regularity. Roughly speaking (see Definition 2.1 below for a precise statement), the -mixing coefficient at lag
is the total variation distance between the actual joint distribution of events separated bytime steps and the product of their marginal distributions, i.e., the distance from independence.
Numerous results in the statistical machine learning literature rely on knowledge of the -mixing coefficients. As Vidyasagar [24, p. 41] notes, -mixing is “just right” for the extension of IID results to dependent data, and so recent work has consistently focused on it. Meir  derives generalization error bounds for nonparametric methods based on model selection via structural risk minimization. Baraud et al.  study the finite sample risk performance of penalized least squares regression estimators under -mixing. Lozano et al.  examine regularized boosting algorithms under absolute regularity and prove consistency. Karandikar and Vidyasagar 
consider “probably approximately correct” learning algorithms, proving that PAC algorithms for IID inputs remain PAC with-mixing inputs under some mild conditions. Ralaivola et al. 
derive PAC bounds for ranking statistics and classifiers using a decomposition of the dependency graph. Finally,Mohri and Rostamizadeh  derive stability bounds for -mixing inputs, generalizing existing stability results for IID data.
All these results assume not just -mixing, but known mixing coefficients. In particular, the risk bounds in [14, 15] and  are incalculable without knowledge of the rates. This knowledge is never available. Unless researchers are willing to assume specific values for a sequence of -mixing coefficients, the results mentioned in the previous paragraph are generally useless when confronted with data.To illustrate this deficiency, consider Theorem 18 of :
Theorem 1.1 (Briefly).
Assume a learning algorithm is -stable. Then, for any sample of size drawn from a stationary -mixing distribution, and
where , has a particular functional form, and is the difference between the true risk and the empirical risk.
Ideally, one could use this result for model selection or to control the size of the generalization error of competing prediction algorithms (support vector machines, support vector regression, and kernel ridge regression are a few of the many algorithms known to satisfy-stability). However the bound depends explicitly on the mixing coefficient . To make matters worse, there are no methods for estimating the -mixing coefficients. According to Meir [14, p. 7], “there is no efficient practical approach known at this stage for estimation of mixing parameters.” We begin to rectify this problem by deriving the first method for estimating these coefficients. We prove that our estimator is consistent for arbitrary -mixing processes. In addition, we derive rates of convergence for Markov approximations to these processes.
Application of statistical learning results to -mixing data is highly desirable in applied work. Many common time series models are known to be -mixing, and the rates of decay are known given the true parameters of the process. Among the processes for which such knowledge is available are ARMA models , GARCH models , and certain Markov processes — see  for an overview of such results. To our knowledge, only Nobel  approaches a solution to the problem of estimating mixing rates by giving a method to distinguish between different polynomial mixing rate regimes through hypothesis testing.
We present the first method for estimating the -mixing coefficients for stationary time series data. Section 2 defines the -mixing coefficient and states our main results on convergence rates and consistency for our estimator. Section 3 gives an intermediate result on the convergence of the histogram estimator with -mixing inputs. Section 4 proves the main results from §2. Section 5 concludes and lays out some avenues for future research.
2 Estimation of -mixing
In this section, we present one of many equivalent definitions of absolute regularity and state our main results, deferring proof to §4.
To fix notation, let
be a sequence of random variables where eachis a measurable function from a probability space into a measurable space . A block of this random sequence will be given by where and are integers, and may be infinite. We use similar notation for the sigma fields generated by these blocks and their joint distributions. In particular, will denote the sigma field generated by , and the joint distribution of will be denoted .
Definition 2.1 (-mixing).
For each positive integer , the the coefficient of absolute regularity, or -mixing coefficient, , is
where is the total variation norm, and is the joint distribution of . A stochastic process is said to be absolutely regular, or -mixing, if as .
Loosely speaking, Definition 2.1 says that the coefficient measures the total variation distance between the joint distribution of random variables seaparted by time units and a distribution under which random variables separated by time units are independent. The supremum over is unnecessary for stationary random processes which is the only case we consider here.
Definition 2.2 (Stationarity).
A sequence of random variables is stationary when all its finite-dimensional distributions are invariant over time: for all and all non-negative integers and , the random vectors and have the same distribution.
Our main result requires the method of blocking used by Yu [26, 27]. The purpose is to transform a sequence of dependent variables into subsequence of nearly IID ones. Consider a sample from a stationary -mixing sequence with density . Let and be non-negative integers such that . Now divide into blocks of each length . Identify the blocks as follows:
be the entire sequence of odd blocks, and let be the sequence of even blocks . Finally, let be a sequence of blocks which are independent of but such that each block has the same distribution as a block from the original sequence:
The blocks are now an IID block sequence, so standard results apply. (See  for a more rigorous analysis of blocking.) With this structure, we can state our main result.
Our main result emerges in two stages. First, we recognize that the distribution of a finite sample depends only on finite-dimensional distributions. This leads to an estimator of a finite-dimensional version of . Next, we let the finite-dimension increase to infinity with the size of the observed sample.
For positive integers , , and , define
where is the joint distribution of . Also, let be the -dimensional histogram estimator of the joint density of consecutive observations, and let be the -dimensional histogram estimator of the joint density of two sets of consecutive observations separated by time points.
We construct an estimator of based on these two histograms.111While it is clearly possible to replace histograms with other choices of density estimators (most notably KDEs), histograms in this case are more convenient theoretically and computationally. See §5 for more details. Define
We show that, by allowing to grow with , this estimator will converge on . This can be seen most clearly by bounding the -risk of the estimator with its estimation and approximation errors:
The first term is the error of estimating with a random sample of data. The second term is the non-stochastic error induced by approximating the infinite dimensional coefficient, , with its -dimensional counterpart, .
Our first theorem in this section establishes consistency of as an estimator of for all -mixing processes provided increases at an appropriate rate. Theorem 2.4 gives finite sample bounds on the estimation error while some measure theoretic arguments contained in §4 show that the approximation error must go to zero as .
Let be a sample from an arbitrary -mixing process. Let where is the Lambert function.222The Lambert function is defined as the (multivalued) inverse of . Thus, is bigger than but smaller than . See for example Corless et al. . Then as .
A finite sample bound for the approximation error is the first step to establishing consistency for . This result gives convergence rates for estimation of the finite dimensional mixing coefficient and also for Markov processes of known order , since in this case, .
Consider a sample from a stationary -mixing process. Let and be positive integers such that and . Then
where and .
Consistency of the estimator is guaranteed only for certain choices of and . Clearly and as are necessary conditions. Consistency also requires convergence of the histogram estimators to the target densities. We leave the proof of this theorem for section 4. As an example to show that this bound can go to zero with proper choices of and , the following corollary proves consistency for first order Markov processes. Consistency of the estimator for higher order Markov processes can be proven similarly. These processes are algebraically -mixing as shown in e.g. Nummelin and Tuominen .
Let be a sample from a first order Markov process with . Then under the conditions of Theorem 2.4, .
Recall that . Then,
if for constants and . In this case, we have that the exponential terms are less than
for and a constant . Therefore, both exponential terms go to 0 as . ∎
Proving Theorem 2.4 requires showing the convergence of the histogram density estimator with -mixing data. We do this in the next section.
3 convergence of histograms
Convergence of density estimators is thoroughly studied in the statistics and machine learning literature. Early papers on the
convergence of kernel density estimators (KDEs) include[25, 2, 21]; Freedman and Diaconis  look specifically at histogram estimators, and Yu  considered the convergence of KDEs for -mixing data and shows that the optimal IID rates can be attained. Devroye and Györfi  argue that is a more appropriate metric for studying density estimation, and Tran  proves consistency of KDEs under - and -mixing. As far as we are aware, ours is the first proof of convergence for histograms under -mixing.
Additionally, the dimensionality of the target density is analogous to the order of the Markov approximation. Therefore, the convergence rates we give are asymptotic in the bandwidth which shrinks as increases, but also in the dimension which increases with . Even under these asymptotics, histogram estimation in this sense is not a high dimensional problem. The dimension of the target density considered here is on the order of , a rate somewhere between and .
If is the histogram estimator based on a (possibly vector valued) sample from a -mixing sequence with stationary density , then for all ,
To prove this result, we use the blocking method of Yu  to transform the dependent -mixing into a sequence of nearly independent blocks. We then apply McDiarmid’s inequality to the blocks to derive asymptotics in the bandwidth of the histogram as well as the dimension of the target density. For completeness, we state Yu’s blocking result and McDiarmid’s inequality before proving the doubly asymptotic histogram convergence for IID data. Combining these lemmas allows us to derive rates of convergence for histograms based on -mixing inputs.
Lemma 3.2 (Lemma 4.1 in Yu ).
Let be a measurable function with respect to the block sequence uniformly bounded by . Then,
where the first expectation is with respect to the dependent block sequence, , and is with respect to the independent sequence, .
This lemma essentially gives a method of applying IID results to -mixing data. Because the dependence decays as we increase the separation between blocks, widely spaced blocks are nearly independent of each other. In particular, the difference between expectations over these nearly independent blocks and expectations over blocks which are actually independent can be controlled by the -mixing coefficient.
Lemma 3.3 (McDiarmid Inequality ).
Let be independent random variables, with taking values in a set for each . Suppose that the measurable function satisfies
whenever the vectors and differ only in the coordinate. Then for any ,
For an IID sample from some density on ,
where is the histogram estimate using a grid with sides of length .
Proof of Lemma 3.4.
Let be the probability of falling into the bin . Then,
For the second claim, consider the bin centered at . Let be the union of all bins . Assume the following:
and is absolutely continuous on , with a.e. partial derivatives
and is absolutely continuous on , with a.e. partial derivatives
for all .
Using a Taylor expansion
where . Therefore, is given by
since the integral of the second term over the bin is zero. This means that for the bin,
Since each bin is bounded, we can sum over all bins. The number of bins is by definition, so
We can now prove the main result of this section.
Proof of Theorem 3.1.
Let be the loss of the histogram estimator, . Here where is the bin containing . Let , , and be histograms based on the block sequences , , and respectively. Clearly Now,
where . Here,
where the probability on the right is for the -field generated by the independent block sequence . Since these blocks are independent, showing that satisfies the bounded differences requirement allows for the application of McDiarmid’s inequality 3.3 to the blocks. For any two block sequences and with for all , then
The proof of Theorem 2.4 relies on the triangle inequality and the relationship between total variation distance and the distance between densities.
Proof of Theorem 2.4.
For any probability measures and defined on the same probability space with associated densities and with respect to some dominating measure ,
Let be the -dimensional stationary distribution of the order Markov process, i.e. in the notation of equation 3. Let be the joint distribution of the bivariate random process created by the initial process and itself separated by time steps. By the triangle inequality, we can upper bound for any . Let and be the distributions associated with histogram estimators and respectively. Then,
where is our estimator and the remaining terms are the distance between a density estimator and the target density. Thus,
A similar argument starting from shows that
so we have that
where and . ∎
The proof of Theorem 2.3 requires two steps which are given in the following Lemmas. The first specifies the histogram bandwidth and the rate at which (the dimensionality of the target density) goes to infinity. If the dimensionality of the target density were fixed, we could achieve rates of convergence similar to those for histograms based on IID inputs. However, we wish to allow the dimensionality to grow with , so the rates are much slower as shown in the following lemma.
For the histogram estimator in Lemma 3.4, let
These choices lead to the optimal rate of convergence.
It is also necessary to show that as grows, . We now prove this result.
converges to as .
By stationarity, the supremum over is unnecessary in Definition 2.1, so without loss of generality, let . Let be the distribution on , and let be the distribution on . Let be the distribution on (the product sigma-field). Then we can rewrite Definition 2.1 using this notation as
Let and be the sub--fields of and consisting of the -dimensional cylinder sets for the dimensions closest together. Let be the product -field of these two. Then we can rewrite as
As such for all and . We can rewrite (11) in terms of finite-dimensional marginals:
where is the restriction of to . Because of the nested nature of these sigma-fields, we have
for all finite . Therefore, for fixed , is a monotone increasing sequence which is bounded above, and it converges to some limit . To show that requires some additional steps.
Let , which is a signed measure on . Let , which is a signed measure on