Reliability of decisions based on tests: Fourier analysis of Boolean decision functions

by   Lourens Waldorp, et al.

Items in a test are often used as a basis for making decisions and such tests are therefore required to have good psychometric properties, like unidimensionality. In many cases the sum score is used in combination with a threshold to decide between pass or fail, for instance. Here we consider whether such a decision function is appropriate, without a latent variable model, and which properties of a decision function are desirable. We consider reliability (stability) of the decision function, i.e., does the decision change upon perturbations, or changes in a fraction of the outcomes of the items (measurement error). We are concerned with questions of whether the sum score is the best way to aggregate the items, and if so why. We use ideas from test theory, social choice theory, graphical models, computer science and probability theory to answer these questions. We conclude that a weighted sum score has desirable properties that (i) fit with test theory and is observable (similar to a condition like conditional association), (ii) has the property that a decision is stable (reliable), and (iii) satisfies Rousseau's criterion that the input should match the decision. We use Fourier analysis of Boolean functions to investigate whether a decision function is stable and to figure out which (set of) items has proportionally too large an influence on the decision. To apply these techniques we invoke ideas from graphical models and use a pseudo-likelihood factorisation of the probability distribution.


page 1

page 2

page 3

page 4


Fourier Entropy-Influence Conjecture for Random Linear Threshold Functions

The Fourier-Entropy Influence (FEI) Conjecture states that for any Boole...

A Fourier View of REINFORCE

We show a connection between the Fourier spectrum of Boolean functions a...

Statistical Analysis of Item Preknowledge in Educational Tests: Latent Variable Modelling and Statistical Decision Theory

Tests are a building block of our modern education system. Many tests ar...

Item Quality Control in Educational Testing: Change Point Model, Compound Risk, and Sequential Detection

In standardized educational testing, test items are reused in multiple t...

Analysis of Boolean Functions

The subject of this textbook is the analysis of Boolean functions. Rough...

The Spearman-Brown Formula and Reliabilities of Random Test Forms

It is shown that the psychometric test reliability, based on any true-sc...

Common Information, Noise Stability, and Their Extensions

Common information (CI) is ubiquitous in information theory and related ...

Please sign up or login with your details

Forgot password? Click here to reset