Nested Conformal Prediction Sets for Classification with Applications to Probation Data

by   Arun K. Kuchibhotla, et al.

Risk assessments to help inform criminal justice decisions have been used in the United States since the 1920s. Over the past several years, statistical learning risk algorithms have been introduced amid much controversy about fairness, transparency and accuracy. In this paper, we focus on accuracy for a large department of probation and parole that is considering a major revision of its current, statistical learning risk methods. Because the content of each offender's supervision is substantially shaped by a forecast of subsequent conduct, forecasts have real consequences. Here we consider the probability that risk forecasts are correct. We augment standard statistical learning estimates of forecasting uncertainty (i.e., confusion tables) with uncertainty estimates from nested conformal prediction sets. In a demonstration of concept using data from the department of probation and parole, we show that the standard uncertainty measures and uncertainty measures from nested conformal prediction sets can differ dramatically in concept and output. We also provide a modification of nested conformal called the localized conformal method to match confusion tables more closely when possible. A strong case can be made favoring the nested and localized conformal approach. As best we can tell, our formulation of such comparisons and consequent recommendations is novel.


page 1

page 2

page 3

page 4


Improving Fairness in Criminal Justice Algorithmic Risk Assessments Using Optimal Transport and Conformal Prediction Sets

In the United States and elsewhere, risk assessment algorithms are being...

Improving Fairness in Criminal Justice Algorithmic Risk Assessments Using Conformal Prediction Sets

Risk assessment algorithms have been correctly criticized for potential ...

The Estimation Risk in Extreme Systemic Risk Forecasts

Systemic risk measures have been shown to be predictive of financial cri...

Multiscale Influenza Forecasting

Influenza forecasting in the United States (US) is complex and challengi...

Branching epistemic uncertainty and thickness of tails

This is an epistemological approach to errors in both inference and risk...

Constraining Localized Vote Tampering in the 2020 US Presidential Election

Voter fraud in the United States is rare and the vote-counting system is...

Checkpointing and Localized Recovery for Nested Fork-Join Programs

While checkpointing is typically combined with a restart of the whole ap...

Please sign up or login with your details

Forgot password? Click here to reset