1 Introduction
A risklimiting audit (RLA) of an election contest is a procedure that has a known minimum chance of leading to a full manual tally of the ballots if the electoral outcome according to that tally would differ from the reported outcome. Outcome means the winner(s) (or, for instance, whether there is a runoff)—not the numerical vote totals. RLAs require a durable, voterverifiable record of voter intent, such as paper ballots, and they assume that this audit trail is sufficiently complete and accurate that a full hand tally would show the true electoral outcome. That assumption is not automatically satisfied: a compliance audit [16] is required to check whether the paper trail is trustworthy.
Current methods for risklimiting audits are generally sequential hypothesis testing procedures: they examine more ballots, or batches of ballots, until either (i) there is strong statistical evidence that a full hand tabulation would confirm the outcome, or (ii) the audit has led to a full hand tabulation, the result of which should become the official result.
RLAs have been conducted in California, Colorado, Indiana, Ohio, Virginia, and Denmark, and are required by law in Colorado (CRS 17515) and Rhode Island (SB 413A and HB 5704A).
The most efficient and transparent sampling design for risklimiting audits selects individual ballots uniformly at random, with or without replacement [13]. Risk calculations for such samples can be made simple without sacrificing rigor [14, 6]. However, to audit contests that cross jurisdictional boundaries then requires coordinating sampling in different counties, and may require different counties to use the lowest common denominator method for assessing risk from the sample, which would not take full advantage of the capabilities of some voting systems. For instance, any system that uses paper ballots as the official record can conduct ballotpolling audits, while ballotlevel comparison audits require systems to generate cast vote records that can be checked manually against a human reading of the paper [5, 6]. (These terms are described in Section 3.)
Stratified RLAs have been considered previously, primarily to conform with legacy audit laws under which counties draw audit samples independently of each other, but also to allow auditors to start the audit before all votebymail or provisional ballots have been tallied, by sampling independently from ballots cast in person, by mail, and provisionally, as soon as subtotals for each group are available [9, 4]
. However, extant methods address only a single approach to auditing, batchlevel comparisons, and only a particular test statistic.
Here, we introduce SUITE, a more general approach to conducting RLAs using stratified samples. SUITE is a twist on intersectionunion tests [7]
, which represent the null hypothesis as the intersection of a number of simpler hypotheses, and the alternative hypothesis as a union of their alternatives. In contrast, here, the null is the union of simpler hypotheses, and the alternative is the intersection of their alternatives. The approach involves finding the maximum
value over a vector of nuisance parameters that describe the simple hypotheses, all allocations of tabulation error across strata for which a full count would find a different electoral outcome than was reported. (A
nuisance parameteris a property of the population that is not of direct interest, but that affects the probability distribution of the data.
Overstatement is error that made the margin of one or more winners over one or more losers appear larger than it really was. The total overstatement across strata determines whether the reported outcome is correct; the overstatements in individual strata are nuisance parameters that affect the distribution of the audit sample.)The basic building block for the method is testing whether the overstatement error in a single stratum is greater than or equal to a quota. Fisher’s combining function is used to merge values for tests in different strata into a single value for the hypothesis that the overstatement in every stratum is greater than or equal to its quota. If that hypothesis can be rejected for all stratumlevel quotas that could change the outcome—that is, if the maximum combined value is sufficiently small—the audit can stop.
It is not actually necessary to consider all possible quotas: the value involves a sum of monotonic functions, which allows us to find upper and lower bounds everywhere using only values on a discrete grid. We present a numerical procedure, implemented in Python, to find bounds on the maximum value when there are two strata. The procedure can be generalized to more than two strata.
Section 2 presents the new approach to stratified auditing. Section 3 illustrates the method by solving a problem pertinent to Colorado: combining ballot polling in one stratum with ballotlevel comparisons in another. This requires straightforward modifications to the mathematics behind ballot polling and ballotlevel comparison to allow the overstatement to be compared to specified thresholds other than the overall contest margin; those modifications are described in Sections 3.1 and 3.2. Section 4 gives numerical examples of simulated audits, using parameters intended to reflect how the procedure would work in Colorado. We provide example software implementing the risk calculations for our recommended approach in Python Jupyter notebooks.^{1}^{1}1See https://github.com/pbstark/CORLA18. Section 5 gives recommendations and considerations for implementation.
2 Stratified audits
Stratified sampling involves partitioning a population into nonoverlapping groups and drawing independent random samples from those groups. [9, 4] developed RLAs based on comparing stratified samples of batches of ballots to hand counts of the votes in those batches: batchlevel comparison RLAs, using a particular test statistic. The method we develop here is more general and more flexible: it can be used with any test statistic, and test statistics in different strata need not be the same—which is key to combining audits of ballots cast using diverse voting technologies.
Here and below, we consider auditing a single plurality contest at a time, although the same sample can be used to audit more than one contest (and supermajority contests), and there are ways of combining audits of different contests into a single process [10, 14]. We use terminology drawn from a number of papers, notably [6].
An overstatement error is an error that caused the margin between any reported winner and any reported loser to appear larger than it really was. An understatement error is an error that caused the margin between every reported winner and every reported loser to appear to be smaller than it really was. Overstatements cast doubt on outcomes; understatements do not, even though they are tabulation errors.
We use to denote a reported winner and to denote a reported loser. The total number of reported votes for candidate is and the total for candidate is . Thus , since is reported to have gotten more votes than .
Let denote the contestwide margin (in votes) of over . We have strata. Let denote the margin (in votes) of reported winner over reported loser in stratum . Note that might be negative in one stratum, but . Let denote the margin (in votes) of reported winner over reported loser that a full hand count would show: the actual margin, in contrast to the reported margin . Reported winner really beat reported loser if and only if . Define to be the actual margin (in votes) of over in stratum .
Let be the overstatement of the margin of over in stratum . Reported winner really beat reported loser if and only if .
An RLA is a test of the hypothesis that the outcome is wrong, that is, that did not really beat : . The null is true if and only if there exists some tuple of real numbers with such that for all .^{2}^{2}2“If” is straightforward. For “only if,” suppose . Set . Then , and for all . Thus if we can reject the conjunction hypothesis at significance level for all such that , we can stop the audit, and the risk limit will be .
2.1 Fisher’s combination method
Fix , with . To test the conjunction hypothesis that stratum null hypotheses are true, that is, that for all , we use Fisher’s combining function. Let be the value of the hypothesis . If the null hypothesis is true, then
(1) 
has a probability distribution that is dominated by the chisquare distribution with
^{3}^{3}3If the stratumlevel tests had continuously distributed values, the distribution would be exactly chisquare with degrees of freedom, but if any of the values has atoms when the null hypothesis is true, it is in general stochastically smaller. This follows from a coupling argument along the lines of Theorem 4.12.3 in [3]. Fisher’s combined statistic will tend to be small when all stratumlevel null hypotheses are true. If any is false, then as the sample size increases, Fisher’s combined statistic will tend to grow.If, for all with , we can reject the conjunction hypothesis at level (i.e., if the minimum value of Fisher’s combined statistic over all is larger than the quantile of the chisquare distribution with degrees of freedom), the audit can stop.
If the audit is allowed to “escalate” in steps, increasing the sample size sequentially, then either the tests used in the separate strata have to be sequential tests, or multiplicity needs to be taken into account, for instance by adjusting the risk limit at each step. Otherwise, the overall procedure can have a risk limit that is much larger than . For examples of controlling for multiplicity when using nonsequential testing procedures in an RLA, see [9, 11].
The stratumlevel value could be a value for the hypothesis from any test procedure. We assume, however, that is based on a onesided test, and that the tests for different values of “nest” in the sense that if , then . This monotonicity is a reasonable requirement because the evidence that the overstatement is greater than should be weaker than the evidence that the overstatement is greater than , if . In particular, this monotonicity holds for the tests proposed in Sections 3.1 and 3.2.
One could use a function other than Fisher’s to combine the stratumlevel values into a value for the conjunction hypothesis, provided it satisfies these properties (see [7]):

the function is nonincreasing in each argument and symmetric with respect to rearrangements of the arguments

the combining function attains its supremum when one of the arguments approaches zero

for every level , the critical value of the combining function is finite and strictly smaller than the function’s supremum.
For instance, one could use Liptak’s function, , or Tippett’s function, .
Fisher’s function is convenient for this application because the tests in different strata are independent, so the chisquared distribution dominates the distribution of when the null hypothesis is true. If tests in different strata were correlated, the null distribution of the combination function would need to be calibrated by simulation; some other combining function might have better properties than Fisher’s [7].
2.2 Maximizing Fisher’s combined value for
We now specialize to strata. The set of such that is then a onedimensional family: if , then . For a given set of data, finding the maximum value over all is thus a onedimensional optimization problem. We provide two software solutions to the problem.
The first approach approximates the maximum via a grid search, refining the grid once the maximum has been bracketed. This is not guaranteed to find the global maximum exactly, although it can approximate the maximum as closely as one desires by refining the mesh, since the objective function is continuous.
The second, more rigorous approach uses bounds on Fisher’s combining function for all . (A lower bound on implies an upper bound on the value: if, for all , the lower bound is larger than the quantile of the chisquared distribution with 4 degrees of freedom, the maximum value is no larger than .)
Some values of can be ruled out a priori, because (for instance) , where is the number of ballots cast in stratum , and thus
(2) 
Let and be lower and upper bounds on .
Recall that are monotonically increasing functions, so, as a function of , increases monotonically and decreases monotonically. Suppose . Then for all , and . Thus
(3) 
This gives a lower bound for on the interval ; the corresponding upper bound is . Partitioning into a collection of intervals and finding and for each yields piecewiseconstant lower and upper bounds for .
If, for all , the lower bound on is larger than the quantile of the chisquare distribution with 4 degrees of freedom, the audit can stop. On the other hand, if for some , the upper bound is less than the quantile of the chisquare distribution with 4 degrees of freedom, or if is less than this quantile at any grid point , the sample size in one or both strata needs to increase. If the lower bound is less than the quantile on some interval, but is above this quantile at every grid point , then one should improve the lower bound by refining the grid and/or by increasing the sample size in one or both strata.
3 Auditing crossjurisdictional contests
As mentioned above, stratified sampling can simplify audit logistics by allowing jurisdictions to sample ballots independently of each other, or by allowing a single jurisdiction to sample independently from different collections of ballots (e.g., votebymail versus cast in person). SUITE allows stratified samples to be combined into an RLA of contests that include ballots from more than one stratum.
We present an example where SUITE is helpful for a different reason: it enables an RLA to take advantage of differences among voting systems to reduce audit sample sizes, which solves a current problem in Colorado.
CRS 17515 requires Colorado to conduct risklimiting audits beginning in 2017. The first risklimiting election audits under this statute were conducted in November, 2017; the second were conducted in July, 2018.^{4}^{4}4See https://www.sos.state.co.us/pubs/elections/RLA/2017RLABackground.html Counties cannot audit contests that cross jurisdictional boundaries (crossjurisdictional contests, such as gubernatorial contests and most federal contests) on their own: margins and risk limits apply to entire contests, not to the portion of a contest included in a county. Colorado has not yet conducted an RLA of a crossjurisdictional contest, although it has performed RLAlike procedures on individual jurisdictions’ portions of some crossjurisdictional contests. To audit statewide elections and contests that cross county lines, Colorado will need to implement new approaches and make some changes to its auditing software, RLATool.
Colorado’s voting systems are heterogeneous. Some counties (containing about 98% of active voters, as of this writing) have voting systems that export cast vote records (CVRs) in a way that the paper ballot corresponding to each CVR can be identified uniquely and retrieved. We call counties with such voting systems CVR counties. In CVR counties, auditors can manually check the accuracy of the voting system’s interpretation of individual ballots. In other counties (legacy or noCVR counties) there is no way to check the accuracy of the system’s interpretation of voter intent for individual ballots.
Contests entirely contained in CVR counties can be audited using ballotlevel comparison audits [6], which compare CVRs to the auditors’ interpretation of voter intent directly from paper ballots. Ballotlevel comparison audits are currently the most efficient approach to risklimiting audits in that they require examining fewer ballots than other methods do, when the outcome of the contest under audit is in fact correct. Contests involving noCVR counties can be audited using ballotpolling audits [5, 6], which generally require examining more ballots than ballotlevel comparison audits to attain the same risk limit.
Colorado’s challenge is to audit contests that include ballots cast in both CVR counties and noCVR counties. There is no literature on how to combine ballot polling with ballotlevel comparisons to audit crossjurisdictional contests that include voters in CVR counties and voters in noCVR counties.^{5}^{5}5See [8] for a different (Bayesian) approach to auditing contests that include both CVR counties and noCVR counties. In general, Bayesian audits are not risklimiting.
Colorado could simply revert to ballotpolling audits for crossjurisdictional contests that include votes in noCVR counties, but that would entail a loss of efficiency. Alternatively, Colorado could use batchlevel comparison audits, with singleballot batches in CVR counties and larger batches in noCVR counties.^{6}^{6}6Since so few ballots are cast in noCVR counties, cruder approaches might work, for instance, pretending that noCVR counties had CVRs, but treating any ballot sampled from a noCVR county as if it had a 2vote overstatement error. See [1]. The statistical theory for such audits has been worked out (see, e.g., [9, 10, 12, 14] and Section 0.A, below); indeed, this is the method that was used in several of California’s pilot audits, including the audit in Orange County, California. However, batchlevel comparison audits were found to be less efficient than ballotpolling audits in these pilots [2].
Moreover, to use batchlevel comparison audits in Colorado would require major changes to RLATool, for reporting batchlevel contest results prior to the audit, for drawing the sample, for reporting audit findings, and for determining when the audit can stop. The changes would include modifying data structures, data uploads, random sampling procedures, and the county user interface. NoCVR counties would also have to revise their audit procedures. Among other things, they would need to report vote subtotals for physically identifiable groups of ballots before the audit starts. NoCVR counties with voting systems that can only report subtotals by precinct might have to make major changes to how they handle ballots, for instance, sorting all ballots by precinct. These are large changes.
We show here that SUITE makes possible a “hybrid” RLA that keeps the advantages of ballotlevel comparison audits in CVR counties but does not require major changes to how noCVR counties audit, nor major changes to RLATool. The key is to use stratified sampling with two strata: ballots cast in CVR counties and those cast in noCVR counties.
In order to use Equation 1, we must develop stratumlevel tests for the overstatement error that are appropriate for the corresponding voting system. Sections 3.1 and 3.2 describe these tests for overstatement in the CVR and noCVR strata, respectively.
3.1 Comparison audits of overstatement quotas
To use comparison auditing in the approach to stratification described above requires extending previous work to test whether the overstatement error is greater than or equal to , rather than simply . Appendix 0.A derives this generalization for arbitrary batch sizes, including batches consisting of one ballot. The derivation considers only a single contest, but the MACRO test statistic [10, 14] automatically extends the result to auditing any number of contests simultaneously. The derivation is for plurality contests, including “votefor” plurality contests. Majority and supermajority contests are a minor modification [9].^{7}^{7}7So are some forms of preferential and approval voting, such as Borda count, and proportional representation contests, such as D’Hondt [15]. For a derivation of ballotlevel comparison risklimiting audits for supermajority contests, see https://github.com/pbstark/S157F17/blob/master/audit.ipynb. (Last visited 14 May 2018.) Changes for IRV/STV are more complicated.
3.2 Ballotpolling audits of overstatement quotas
To use the new stratification method with ballot polling requires a different approach than [5] took: their approach tests whether got a larger share of the votes than , but we need to test whether the margin in votes in the stratum is greater than or equal to a threshold (namely, ). This introduces a nuisance parameter, the number of ballots with votes for either or . We address this by maximizing the probability ratio in Wald’s Sequential Probability Ratio Test [17] over all possible values of the nuisance parameter. Appendix 0.B develops the test.
4 Numerical examples
Jupyter notebooks containing calculations for hybrid stratified audits intended to be relevant for Colorado are available at https://www.github.com/pbstark/CORLA18.
hybridauditexample1 contains two hypothetical examples. The first has cast ballots, of which 9.1% were in noCVR counties. The diluted margin (the margin in votes, divided by the total number of ballots cast) is . In 94% of 10,000 simulations in which the reported results were correct, drawing 700 ballots from the CVR stratum and 500 ballots from the noCVR stratum (1,200 ballots in all) allowed SUITE to confirm the outcome at 10% risk. For the remaining 6%, further expansion of the audits would have been necessary.
If it were possible to conduct a ballotlevel comparison audit for the entire contest, an RLA with risk limit 10% could terminate after examining 263 ballots if it found no errors. A ballotpolling audit of the entire contest would have been expected to examine about 14,000 ballots, more than 10% of ballots cast. The hybrid audit is less efficient than a ballotlevel comparison audit, but far more efficient than a ballotpolling audit.
The second contest has 2 million cast ballots, of which 5% were cast in noCVR counties. The diluted margin is about . The workload for SUITE at 5% risk is quite low: In 100% of 10,000 simulations in which the reported results were correct, auditing 43 ballots from the CVR stratum and 15 ballots from the noCVR stratum would have confirmed the outcome. If it were possible to conduct a ballotlevel comparison audit for the entire contest, an RLA at risk limit 5% could terminate after examining 31 ballots if it found no errors. The additional work for the hybrid stratified audit is disproportionately in the noCVR counties.
A second notebook, hybridauditexample2, illustrates the workflow for SUITE for an election with 2 million ballots cast. The reported margin is just over , but the reported winner and reported loser are actually tied in both strata. The risk limit is 5%. For a sample of 500 ballots from the CVR stratum and 1000 ballots from the noCVR stratum, the maximum combined value is over 25%, so the audit cannot stop there.
A third notebook, fisher_combined_pvalue, illustrates the numerical methods used to check whether the maximum combined value is below the risk limit. It includes code for the tests in the two strata, for the lower and upper bounds and for , for evaluating Fisher’s combining function on a grid, and for computing bounds on the value via Equation 3.
5 Discussion
We present SUITE, a new class of procedures for RLAs based on stratified random sampling. SUITE is agnostic about the capability of voting equipment in different strata, unlike previous methods, which require batchlevel comparisons in every stratum. SUITE allows arbitrary tests to be used in different strata; if those tests are sequentially valid, then the overall RLA is sequential. (Otherwise, multiplicity adjustments might be needed if one wants an audit that escalates in stages. See [9, 11] for two approaches.)
Like other RLA methods, SUITE poses auditing as a hypothesis test. The null hypothesis is a union over all partitions of outcomechanging error across strata. The hypothesis is rejected if the maximum value over all such partitions is sufficiently small. Each possible partition yields an intersection hypothesis, tested by combining values from different strata using Fisher’s combining function (or a suitable replacement).
Among other things, the new approach solves a current problem in Colorado: how to conduct RLAs of contests that cross jurisdictional lines, such as statewide contests and many federal contests.
We give numerical examples in Jupyter notebooks that can be modified to estimate the workload for different contest sizes, margins, and risk limits. In our numerical experiments, the new method requires auditing far fewer ballots than previous approaches would.
Appendix 0.A Comparison tests for an overstatement quota
0.a.1 Notation

: the set of reported winners of the contest

: the set of reported losers of the contest

ballots were cast in stratum . (The contest might not appear on all ballots.)

“batches” of ballots are in stratum . A batch contains one or more ballots. Every ballot in stratum is in exactly one batch.

: number of ballots in batch . .

: reported votes for candidate in batch

: actual votes for candidate in batch . If the contest does not appear on any ballot in batch , then .

: Reported margin in stratum of reported winner over reported loser , in votes.

: overall reported margin in votes of reported winner over reported loser for the entire contest (not just stratum )

: smallest reported overall margin in votes between any reported winner and reported loser

: actual margin in votes in the stratum of reported winner over reported loser

: actual margin in votes of reported winner over reported loser for the entire contest (not just in stratum )
0.a.2 Reduction to maximum relative overstatement
If the contest is entirely contained in stratum , then the reported winners of the contest are the actual winners if
Here, we address the case that the contest may include a portion outside the stratum. To combine independent samples in different strata, it is convenient to be able to test whether the net overstatement error in a stratum is greater than or equal to a given threshold.
Instead of testing that condition directly, we will test a condition that is sufficient but not necessary for the inequality to hold, to get a computationally simple test that is still conservative (i.e., the level is not larger than its nominal value).
For every winner, loser pair , we want to test whether the overstatement error is greater than or equal to some threshold, generally one tied to the reported margin between and . For instance, for a hybrid stratified audit, we set the threshold to be .
We want to test whether
The maximum of sums is not larger than the sum of the maxima; that is,
Define
Then no reported margin is overstated by a fraction or more if
Thus if we can reject the hypothesis , we can conclude that no pairwise margin was overstated by as much as a fraction .
Testing whether would require a very large sample if we knew nothing at all about without auditing batch : a single large value of could make arbitrarily large. But there is an a priori upper bound for . Whatever the reported votes are in batch , we can find the potential values of the actual votes that would make the error largest, because must be between 0 and , the number of ballots in batch :
Hence,
(4) 
Knowing that might let us conclude reliably that by examining only a small number of batches—depending on the values and on the values of for the audited batches.
To make inferences about , it is helpful to work with the taint . Define . Suppose we draw batches at random with replacement, with probability of drawing batch in each draw, . (Since , these are all positive numbers, and they sum to 1, so they define a probability distribution on the batches.)
Let be the value of for the batch selected in the th draw. Then are IID, , and
Thus . So, if we have strong evidence that , we have strong evidence that .
This approach can be simplified even further by noting that has a simple upper bound that does not depend on . At worst, the reported result for batch shows votes for the “leastwinning” apparent winner of the contest with the smallest margin, but a hand interpretation would show that all ballots in the batch had votes for the runnerup in that contest. Since and ,
Thus if we use in lieu of , we still get conservative results. (We also need to redefine to be the sum of those upper bounds.) An intermediate, still conservative approach would be to use this upper bound for batches that consist of a single ballot, but use the sharper bound (4) when . Regardless, for the new definition of and , are IID, , and
So, if we have evidence that , we have evidence that .
0.a.3 Testing
A variety of methods are available to test whether . One particularly elegant sequential method is based on Wald’s Sequential Probability Ratio Test (SPRT) [17]. Harold Kaplan pointed out this method on a website that no longer exists. A derivation of this KaplanWald method is in Appendix A of [15]; to apply the method here, take in their equation 18. A different sequential method, the KaplanMarkov method (also due to Harold Kaplan), is given in [12].
Appendix 0.B Ballotpolling tests for an overstatement quota
In this section, we derive a ballotpolling test of the hypothesis that the margin (in votes) in a single stratum is greater than or equal to a threshold .
0.b.1 Wald’s SPRT with a nuisance parameter
Consider a single stratum containing ballots, of which have a vote for but not for , have a vote for but not for , and have votes for both and or neither nor , including undervotes and invalid ballots. Ballots are drawn sequentially without replacement, with equal probability of selecting each asyetunselected ballot in each draw.
We want to test the compound hypothesis that against the alternative that , , and , with .
The values , , and are the reported results for stratum (or values related to those reported results; see [5]). In this problem, (equivalently, ) is a nuisance parameter: we care about .
Let be , , or according to whether the ballot selected on the th draw shows a vote for but not , but not , or something else. Let ; and define and analogously.
The probability of a given data sequence under the alternative hypothesis is
If , the data obviously do not provide evidence against the null, so we suppose that , in which case, the element of the null that will maximize the probability of the observed data has . Under the null hypothesis, the probability of is
for some value and the corresponding . How large can that probability be under the null? The probability under the null is maximized by any integer that maximizes
The logarithm is monotonic, so any maximizer also maximizes
The first two terms on the right increase monotonically with and the last term decreases monotonically with . This yields bounds without having to evaluate everywhere. Suppose . Then for all integer between and ,
The optimization problem can be solved using a branch and bound approach. For instance, start by evaluating
and
at , , and their midpoint, to get the values of at those three points, along with upper bounds on on the ranges between them. At stage , we have evaluated , , and at points , and we have upper bounds on on the ranges between those points. Let be the upper bound on for . Suppose that for some , . Then is a global maximizer of . If there is some , then subdivide the range with the largest , calculate , , and at the new point, and repeat. This algorithm must terminate by identifying a global maximizer after a finite number of steps.
A conservative value for the null hypothesis after items have been drawn is thus
Because the test is built on Wald’s SPRT, the sample can expand sequentially and (if the null hypothesis is true) the chance that is never larger than . That is, if the null is true.
A Jupyter notebook implementing this approach is given in https://github.com/pbstark/CORLA18.
References
 [1] Bañuelos, J., Stark, P.: Limiting risk by turning manifest phantoms into evil zombies. Tech. rep., arXiv.org (2012), http://arxiv.org/abs/1207.3413, retrieved 17 July 2012
 [2] California Secretary of State: California Secretary of State PostElection RiskLimiting Audit Pilot Program 20112013: Final Report to the United States Election Assistance Commission. http://votingsystems.cdn.sos.ca.gov/oversight/riskpilot/finalreport073014.pdf Retrieved 6 May 2018 (2014)
 [3] Grimmett, G.R., Stirzaker, D.R.: Probability and Random Processes. Oxford University Press (August 2001), http://www.amazon.ca/exec/obidos/redirect?tag=citeulike0920&path=ASIN/0198572220
 [4] Higgins, M., Rivest, R., Stark, P.: Sharper pvalues for stratified postelection audits. Statistics, Politics, and Policy 2(1) (2011), http://www.bepress.com/spp/vol2/iss1/7
 [5] Lindeman, M., Stark, P., Yates, V.: BRAVO: Ballotpolling risklimiting audits to verify outcomes. In: Proceedings of the 2011 Electronic Voting Technology Workshop / Workshop on Trustworthy Elections (EVT/WOTE ’11). USENIX (2012)
 [6] Lindeman, M., Stark, P.B.: A gentle introduction to risklimiting audits. IEEE Security and Privacy 10, 42–49 (2012)
 [7] Pesarin, F., Salmaso, L.: Permutation tests for complex data: Theory, applications, and software. John Wiley and Sons, Ltd., West Sussex, UK (2010)
 [8] Rivest, R.L.: Bayesian tabulation audits: Explained and extended (January 1, 2018), https://arxiv.org/abs/1801.00528
 [9] Stark, P.: Conservative statistical postelection audits. Ann. Appl. Stat. 2, 550–581 (2008), http://arxiv.org/abs/0807.4005
 [10] Stark, P.: Auditing a collection of races simultaneously. Tech. rep., arXiv.org (2009), http://arxiv.org/abs/0905.1422v1
 [11] Stark, P.: CAST: Canvass audits by sampling and testing. IEEE Transactions on Information Forensics and Security, Special Issue on Electronic Voting 4, 708–717 (2009)
 [12] Stark, P.: Risklimiting postelection audits: values from common probability inequalities. IEEE Transactions on Information Forensics and Security 4, 1005–1014 (2009)
 [13] Stark, P.: Risklimiting votetabulation audits: The importance of cluster size. Chance 23(3), 9–12 (2010)
 [14] Stark, P.: Supersimple simultaneous singleballot risklimiting audits. In: Proceedings of the 2010 Electronic Voting Technology Workshop / Workshop on Trustworthy Elections (EVT/WOTE ’10). USENIX (2010), http://www.usenix.org/events/evtwote10/tech/full_papers/Stark.pdf
 [15] Stark, P.B., Teague, V.: Verifiable european elections: Risklimiting audits for d’hondt and its relatives. JETS: USENIX Journal of Election Technology and Systems 3.1 (2014), https://www.usenix.org/jets/issues/0301/stark
 [16] Stark, P.B., Wagner, D.A.: Evidencebased elections. IEEE Security and Privacy 10, 33–41 (2012)
 [17] Wald, A.: Sequential tests of statistical hypotheses. Ann. Math. Stat. 16, 117–186 (1945)
Comments
There are no comments yet.