Monte Carlo Sampling Bias in the Microwave Uncertainty Framework

by   Michael Frey, et al.

Uncertainty propagation software can have unknown, inadvertent biases introduced by various means. This work is a case study in bias identification and reduction in one such software package, the Microwave Uncertainty Framework (MUF). The general purpose of the MUF is to provide automated multivariate statistical uncertainty propagation and analysis on a Monte Carlo (MC) basis. Combine is a key module in the MUF, responsible for merging data, raw or transformed, to accurately reflect the variability in the data and in its central tendency. In this work the performance of Combine's MC replicates is analytically compared against its stated design goals. An alternative construction is proposed for Combine's MC replicates and its performance is compared, too, against Combine's design goals. These comparisons are made within an archetypal two-stage scenario in which received data are first transformed in conjunction with shared systematic error and then combined to produce summary information. These comparisons reveal the limited conditions under which Combine's uncertainty results are unbiased and the extent of these biases when these conditions are not met. For small MC sample sizes neither construction, current or alternative, fully meets Combine's design goals, nor does either construction consistently outperform the other. However, for large MC sample sizes the bias in the proposed alternative construction is asymptotically zero, and this construction is recommended.



There are no comments yet.



Modern Monte Carlo Methods for Efficient Uncertainty Quantification and Propagation: A Survey

Uncertainty quantification (UQ) includes the characterization, integrati...

Reducing the error of Monte Carlo Algorithms by Learning Control Variates

Monte Carlo (MC) sampling algorithms are an extremely widely-used techni...

Adaptive Newton-Monte Carlo for efficient and fully error controlled yield optimization

In this paper we present an efficient and fully error controlled algorit...

Density estimation by Randomized Quasi-Monte Carlo

We consider the problem of estimating the density of a random variable X...

Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates

We consider the core reinforcement-learning problem of on-policy value f...

A Generalized Framework for Approximate Control Variates

We describe and analyze a Monte Carlo (MC) sampling framework for accele...

Approximating Gaussian Process Emulators with Linear Inequality Constraints and Noisy Observations via MC and MCMC

Adding inequality constraints (e.g. boundedness, monotonicity, convexity...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Modern national, industrial, and academic laboratories engaged in high-precision metrology rely on statistical software for multivariate and functional measurement uncertainty propagation and analysis. This software is typically highly complex and flexible and often has a Monte Carlo basis. Even in software well-designed from a statistical perspective, biases can be inadvertantly introduced due variously to flaws in statistical procedures, the algorithms that support them, or the algorithms’ coding. Statistical experiments are a natural and powerful way to test for such biases. We report the results of a case study of a microwave measurement uncertainty software package, called the Microwave Uncertainty Framework (MUF), in which a significant heretofore unknown bias in the software was detected, characterized, and corrected. This case study shows that elementary statistical performance testing can successfully identify such biases.

State-of-the-art microwave measurement relies on high-speed instrumentation including vector network analyzers (VNAs) operating in the frequency domain, temporal sampling oscilloscopes, and an array of other instruments, often used simultaneously in the same experiment. The refined measurements made possible by these arrangements allow investigators to, for example, identify the multiple reflections created by small imperfections in microwave systems, capture distortions due to the systems’ frequency-limited electronics, and study the role of noise. Statistical analysis of the data from this mix of instrumentation, including the conduct of uncertainty analyses, often involves shifts between the time and frequency domains. These shifts require that microwave uncertainty analyses account, particularly, for statistical correlations among the measurement uncertainties. To see this, consider that imperfections in microwave systems are often the source of unwanted reflections and attendant power losses. These temporal effects Fourier-map into the frequency domain as ripples with a characteristic period related to the inverse of the reflections’ time spacing. The VNA is currently the most accurate instrument for measuring these multiple reflections, and the errors made by this frequency sampling instrument typically manifest themselves as correlated time domain errors in the magnitudes, shapes, and positions of the multiple reflections. Statistical uncertainties in VNA measurements cannot be transformed correctly into the time domain without accounting for correlations created by the domain transformation


Microwave measurement instrumentation with its often voluminous data production and the data-analytic need to track statistical correlations have motivated three automated approaches for statistical uncertainty analysis, the METAS VNA Tools II software package [3], the Garelli-Ferrero (GF) software package [4], and the MUF. The METAS and GF software packages support microwave multi-port network investigations, and offer fast, efficient sensitivity analysis implementations [5].

The MUF is a software suite created, supported, and made publicly available by the Radio Frequency Division of the U.S. National Institute of Standards and Technology. The MUF has capabilities similar to those of the METAS and GF software packages, while supporting radio frequency engineering applications beyond network analysis. The MUF’s general purpose is to provide automated multivariate statistical type A- and type B-evaluated [6] uncertainty propagation and analysis on a Monte Carlo (MC) basis [7] accessible through user-friendly interfaces. The MUF’s MC capability preserves non-Gaussian features of measured multivariate microwave signals, identifying systematic biases, for example, in signal calibration and processing steps. The MUF is composed of functional modules selectable by the user as needed for analyses. These include Model modules to flexibly represent microwave system elements. Model modules are useful, for example, for building calibration models, and they can be cascaded to represent increasingly complex systems. Other processing modules, termed here Transform

modules, are available to perform oscilloscope and receiver calibrations, Fourier transforms, and other user-defined custom analytical transformations.

Combine is a key module in the MUF, responsible for merging data, raw or transformed, to accurately reflect the variability in the data and in its central tendency. Combine is designed to be used at any point in extended analyses where repeated measurements must be merged. This flexibility is a powerful feature of the MUF.

This paper presents an analysis of the Combine module in the MUF. Because of the MUF’s distributed, multi-user, multi-purpose nature, Combine can be executed at different stages of uncertainty propagation analysis. We study a common use of Combine described by the two-stage scenario diagrammed in Fig. 1. In the first stage of this scenario, multivariate data are joined with shared systematic error in a bank of Transform modules and then, in the second stage, the transformed data are combined within the Combine module. Transform’s outputs take the form of nominal values of a selected mathematical transformation with associated uncertainties. Transform’s outputs in this form allow us in our two-stage scenario to study and assess Combine’s operation in which its inputs with their associated uncertainties are used to produce a summary mean output with an associated uncertainty. Combine represents the uncertainty in the summary mean in various fashions but provides the most detail in the form of a sample of MC replicates. Our analysis of Combine focuses specifically on the bias in the mean and covariance of these MC replicates. This analysis reveals that Combine’s construction of MC replicates is fundamentally biased, and we propose an alternative construction that effectively eliminates this bias.

Figure 1. Two-stage MUF use scenario in which data vectors are transformed in conjunction with MC-generated shared systematic error to produce nominal values and corresponding MC replicates . These results are then merged in the Combine module to produce a combined nominal value with MC replicates .

The remainder of the paper is organized as follows. In Sect. 2 we analyze Combine’s performance in the two-stage scenario diagrammed in Fig. 1, showing that the sample mean of the MC replicates has zero bias and giving an analytical expression for the bias in the covariance of its MC replicates. This covariance bias is studied for specific cases of additive, multiplicative, exponential, and phase error. In Sect. 3 we propose an alternative construction for Combine’s MC replicates and show that the sample mean of Combine

’s MC replicates has zero bias. We put tight bounds on the corresponding covariance bias and show that this bias is asymptotically zero; in this latter regard the proposed construction is better than the current method. Estimation bias is the primary concern in MC sampling, but MC estimation variability is also an issue. In Sect. 4 we continue our comparison of the current and alternative MC replicate constructions, comparing the variability in their sample means and sample covariances. We conclude in Sect. 5 with summary remarks supporting adoption of the proposed alternative MC replicate construction method in place of


’s current method. For the results presented in the following sections, we assume without note that the usual technical conditions pertain, that functions are measurable, that moments of sufficient order exist, etc.

2 Bias in the Combine module

We suppose in the two-stage scenario in Fig. 1 that the data vectors (of length ) are identically distributed and mutually independent and write to identify the mean and covariance matrix of . We also suppose that the MC-generated, length- errors , are identically distributed, mutually independent, and independent of the sample of data vectors . The mean and covariance of are . The covariance represents random uncertainty in the measurement of while the errors are systematic post-measurement errors introduced among the due, for example, to calibration adjustments.

Each data vector in Fig. 1 is operated on individually by Transform, producing for a nominal value


for and a sample of vector Monte Carlo (MC) replicates


for . The superscripts in (1) and (2) signify that these are Transform outputs in the first stage in our scenario. The MC replicates in (2) vary for a given only according to random replicates from the distribution of . The same random replicates are used to create each ’s sample of MC replicates. This models systematic errors that are shared among the . The Transform module can similarly implement unshared systematic errors by using independent sets of for each , but the need for this capability rarely arises in application.

The transformation in Transform has the general form


where and are the th components of the vectors and , respectively. The scalar-valued function is a user-specified parameter in Transform. Some choices of are , , , and , representing additive, multiplicative, phase, and exponential error, respectively. Transform also has many optional parameters, among them two matrix parameters, and . When, for example, the user specifies a matrix value for , is applied to instead of . The matrix parameter operates similarly. Mathematically, and in the specification of are redundant; for example, we can without loss of generality take by substituting and for and , respectively. The optional use of , though, gives Transform representational flexibility, allowing it, for example, to implement the Fourier transform of . The covariance of is interpreted as the error covariance associated with measurement of , so and reflect unrelated physical processes and have distinct modeling roles. The matrices and play similarly distinct modeling roles.

Combine in Fig. 1’s two-stage scenario produces a nominal value for the transformed data and a sample of MC replicates to describe the distribution and, particularly, the uncertainty of the transformed and combined data. These Combine outputs are given by




where and , . Further, the are independent of both the and . The matrices and are the unitary and diagonal members, respectively, of the eigendecomposition of the sample covariance matrix


associated with the nominal vectors at Combine’s input.

The vectors in (5) model standard normal variation along the principal axes of

. These standard normal variates are scaled by the standard deviations in

and then rotated by onto the coordinate axes of to reflect the variability of the transformed data at Combine’s input. The components of the

are chosen to be normally distributed based on the assumption that the number

of Combine

inputs and their independence are together great enough to support a Central Limit Theorem approximation. We present the

as normally distributed because this is how they are generated in Combine. Only in subsection 4.2, however, is this distributional assumption necessary to our results.

The nominal value in (4) produced by Combine

is a natural, intuitive summary of the central tendency of the transformed data provided that the transformed data are unimodal with little skew. The purpose of the MC replicates

is to indicate central tendency under more general conditions as well as to summarize the spread and distributional shape of the estimated central tendency. Formally, Combine is designed to produce a sample of MC replicates whose mean

is an unbiased estimator of the vector

and whose covariance


is an unbiased estimator of the covariance of the vector . In other words, the MC replicates in (5) should satisfy




We note for later use that under the conditions of our two-stage scenario the estimands in (8) and (9) can be expressed as



with . Our analysis, summarized in Proposition 1 below, of the MC construction in (5) shows that Combine meets design goals (8) and (9) only under certain conditions, and that without these conditions Combine exhibits bias.

Proposition 1: Suppose that, in the two-stage scenario in Fig. 1 , we have independent, identically distributed data vectors . Also suppose we have independent, identically distributed errors . Assume the sets of and are independent. Suppose further that the Transform outputs and are given by (1) and (2) with as in (3), and the Combine outputs and are given by (4) and (5). Then




where is the difference of two covariances


Proposition 1 establishes that the design goal in (8) is generally met by the MC replicates in (5), but the design goal in (9) is not. The covariance in the sample of MC replicates is biased by an amount . We will see in the next section that this bias can be positive or negative. We first prove the two parts (12) and (13) of the proposition.

Proof of (12): We first note that since the are identically distributed. Then, conditioning on the factor in (5) and using that and are independent, we have

Since , this yields , which proves (12).

To prove (13) in the proposition, we need four lemmas, which we state here. Their proofs are given in the appendix. Lemma 1 concerns the sample covariance of cross-correlated vectors. Lemmas 2 and 3 are elementary conditioning argument-based results for auto- and cross-covariances. Lemma 4 is used here and in the proofs of subsequent propositions.

Lemma 1: Let for , with sample covariance

Let be the cross-covariance of and , and suppose the are cross-correlated with for all . Then .

Lemma 2: Let be independent of the vector-matrix pair . Then .

Lemma 3: Let , and suppose , , and are mutually independent. Then .

Lemma 4: Let be independent, identically distributed random vectors independent of the random vector . Let be a vector function of and . Then .

Proof of (13): The MC replicate vectors created by Combine are correlated with common cross-covariance . Therefore, according to Lemma 1,


with . Using definition (5) for , Lemma 2, definition (2) for , and the eigendecomposition , we have


The Transform nominal values in (1) are independent and identically distributed so


and (16) becomes


Now consider the cross-covariance in (15). Using definition (5) for , Lemma 3, and definition (2) for , we have


the last equality holding because the data vectors are independent. Applying Lemma 4 to the covariance in (19) and substituting the result along with (18) back into (15) proves (13).

2.1 Example error models

Proposition 1’s point is that the MC replicates produced by Combine in our two-stage scenario have a covariance bias . In the remainder of this section we evaluate for various error models, showing that can be positive, negative, or zero. Where is non-zero, we show in the univariate case that the relative bias


approaches 20% in one example and even 200% in another.

Additive error: The function in (3) is for additive error. In this case and

so in (14) is identically zero. Thus for additive shared systematic error Combine’s MC replicates have both zero mean bias and zero covariance bias.

Multiplicative error: The function in (3) is for multiplicative error and the th component of is where and are the th components of and . We have

Therefore and . This shows that for multiplicative shared systematic error Combine’s MC replicates have both zero mean bias and zero covariance bias.

Phase error: The function in (3) is for phase error. In this case the covariance in Combine’s MC replicates can be biased. We focus on the univariate case in which we have scalars, and

, and the phase error is uniformly distributed,

, , with mean and range . We note first that


and, using ,

To assess the relative size of the bias associated with above, we consider the extremal case where and where is

with equal probabilities. Then

, , and for . Using (2), we find that the relative bias (20

) associated with the MC sample variance is

Here the relative bias is 200% for any sample size . This albeit exteme example demonstrates that very large relative biases are possible with Combine’s current method of MC replicate construction.

Exponential error: The function in (3) is for exponential error. In this case the covariance in Combine’s MC replicates can be positively or negatively biased. We focus on the case of uniformly distributed scalars, and . For this case we find that is broadly, but not always, negative.

Let and , . We have so in (14) is where

Using and evaluating numerically, we find that is slightly positive for small , as shown in Fig. 2. Otherwise, in the region , is negative, increasingly so for larger ranges and .

In the cases presented in Fig. 2 for exponential error, the covariance

is positive. According to (2), then, the relative bias (20) associated with is strongest at the smallest sample size , in which case

Numerical evaluation of this expression yields the results presented in Fig. 3. At its strongest the relative bias approaches 20% for .

Figure 2. Bias factor with exponential error. The data and the exponential error are both uniformly distributed, and . The bias factor is negative everywhere except in the small blue patches where .

Figure 3. Relative bias of Combine’s MC sample variance with exponential error. The data and the exponential error are both uniformly distributed, and . The relative bias approaches 20% for .

3 An alternative MC construction

The previous section shows that Combine’s MC replicates in (5) generated for the two-stage scenario in Fig. 1 fail to fully meet Combine’s design goals (8) and (9). We propose in this section an alternative construction for Combine’s MC replicates. Like the replicates in (5), the proposed replicates meet goal (8). Unlike the replicates, the replicates essentially meet goal (9), doing so arbitrarily closely for sufficiently large MC replicate sample size .



where and , . Further, the are independent of both the and . In this alternative construction the matrices and are now the unitary and diagonal members, respectively, of the eigendecomposition of the sample covariance


associated with the means of the MC samples at Combine’s input. Proposition 2 below shows that basing the sample variability of the stage-two Combine MC replicates on the stage-one MC means instead of on the stage-one nominal values essentially removes the bias identified in Proposition 1. This reduced bias is explained in some part by the greater information retained by using the MC means instead of the nominal values: the reflect nonlinearities in across the full distribution of , while the nominal values are only exposed to at the mean of the distribution.

Proposition 2: Let the set-up be the same as in Proposition 1 except that the Combine-stage MC replicates in Fig. 1 are given by in (21). Then




where is the difference of two covariances


Proof: The proof of (23) is the same as that of (12) because and in (21) are again independent. To prove (24), we first note that the arguments based on Lemmas 1, 2, and 3 early in the proof of (13) apply also here, giving






in which case


Using Lemma 1, we write in (27) as