## 1 Introduction

Users voluntarily provide huge amounts of personal data to online services, such as Facebook, Google, and Amazon, in exchange for useful services. But a basic principle of data autonomy asserts that users should be able to revoke access to their data if they no longer find the exchange of data for services worthwhile. Indeed, each of these organizations provides a way for users to request that their data be deleted. This is related to, although distinct from the “Right to be Forgotten” from the European Union’s General Data Protection Act (GPDR). The Right to be Forgotten entails the right for users, in certain circumstances, to request that negative information *concerning* them to be removed. Like basic data autonomy, it sometimes obligates companies to delete data.

But what does it mean to delete data? Typically, user data does not sit siloed in a database, but rather is used to produce derivatives such as predictive models. Deleting a user’s data from a database may prevent it from influencing the training of future models^{1}^{1}1Or perhaps not, if previously trained models (trained before a user’s data deletion) are used as inputs to the subsequent models., but does not remove the influence of a user’s data on existing models — and that influence may be significant. For example, it is possible to extract information about specific data points used for training from models that have been trained in standard ways [attack]. So deleting a user’s data naively, by simply removing it from a database, may not accomplish much: what we really want is to remove (or at least rigorously limit) the *influence* that an individual’s data has on the behavior of any part of the system.

How should we accomplish this? We could *retrain* all predictive models from scratch every time a user requests that their data be removed, but this would entail an enormous computational cost. Ginart et al. [forgetu] propose a compelling alternative: full retraining is unnecessary if we can design a deletion operation that produces a (distribution of) model output(s) that is statistically indistinguishable from the (distribution of) model output(s) that would have arisen from full retraining. Ginart et al. [forgetu] also propose an approximate notion of deletion that uses a differential-privacy like measure of “approximate” statistical indistinguishability that we adopt in this work.

### 1.1 Our Results and Techniques

In this paper, we consider *convex* models that are trained to some specified accuracy, and then are deployed while a sequence of requests arrive to delete (or add) additional data points. The deletion or addition must happen immediately, before the next point comes in, using only a fixed running time (which we measure in terms of gradient computations) per update. We require that the distribution on output models be -indistinguishable from the distribution on output models that would result from full retraining (see Section LABEL:sec:model for the precise definition: this is a notion of approximate statistical indistinguishability from the differential privacy literature). In a departure from prior work, we make the distinction between whether the entire *internal state* of the algorithm must be indistinguishable from full re-training, or whether we only require statistical indistinguishability with respect to the *observable outputs* of the algorithms. If we require indistinguishability with respect to the full internal state, we call these update or *unlearning* algorithms *perfect*. This is similar to the distinction made in the differential privacy literature, which typically only requires indistinguishability for the *outputs* of private algorithms, but which has a strengthening (called *pan privacy* [pan1, pan2]) which also requires that the internal state satisfy statistical indistinguishability. We remark that while unlearning algorithms that are allowed to maintain a “secret state” that need not satisfy the data deletion notion require additional trust in the *security* of the training system, this is orthogonal to *privacy*. Indeed, [DifferencingPrivacy] show that even without secret state, algorithms satisfying standard deletion guarantees can exacerbate membership inference attacks if the attacker can observe the model both before and after a deletion (because standard deletion guarantees promise nothing about what can be learned about an individual from two model outputs). In contrast, although some of our unlearning algorithms maintain a secret state that does not satisfy the statistical indistinguishability property, our model outputs themselves satisfy -differential privacy. This in particular prevents membership inference attacks from observers who can observe a small number of output models, so long as they cannot observe the secret state. All prior work has focused on perfect unlearning.

We introduce another novel distinction between *strong* unlearning algorithms and *weak* unlearning algorithms. For an unlearning algorithm to be *strong*, we require that for a fixed accuracy target, the run-time of the update operation be constant (or at most logarithmic) in the length of the update sequence. A weak unlearning algorithm may have run-time per update (or equivalently, error) that grows polynomially with the length of the update sequence. All prior work has given weak unlearning algorithms.

We give two sets of results. The first, which operates under the most permissive set of assumptions, is a simple family of gradient descent algorithms. After each addition or deletion request, the update algorithm starts from the previous model, and performs a small number of gradient descent updates — sufficient to guarantee that the model parameter is boundedly close to the *optimal* model parameter in Euclidean distance. It then perturbs the model parameter with Gaussian noise of sufficient magnitude to guarantee

-indistinguishability with respect to anything within a small neighborhood of the optimal model. We prove that this simple approach yields a strong, perfect unlearning algorithm for loss functions that are strongly convex and smooth. Without the strong convexity assumption, we can still derive strong unlearning algorithms, but ones which must maintain secret state. We can further improve our accuracy guarantees if we are willing to settle for weak unlearning algorithms. The per-round computation budget and the achievable steady state accuracy can be smoothly traded off against one another.

Our second algorithm improves over the straightforward approach above (under slightly stronger regularity assumptions) when the data dimension is sufficiently large. It first takes a bootstrap sample from the underlying dataset, and then randomly partitions it into

parts. The initial training algorithm separately and independently optimizes the loss function on each part, and then averages the parameter vector from each part, before finally releasing the perturbed average. Zhang et al

[ZDW12] analyzed this algorithm (absent the final perturbation) and proved accuracy bounds with respect to the underlying distribution (which for us is the dataset from which we draw the bootstrap sample). Our update operation involves first using a variant of reservoir-sampling that maintains the property that the union of the partitions continue to be distributed as independent samples drawn with replacement from our current dataset. We then use the simple gradient based update algorithms from our first set of results to update the parameters*only from the partitions that have been modified by the addition or deletion*. Because each of these partitions contains only a fraction of the dataset, we can use our fixed gradient computation budget to perform more updates. Because we have maintained the marginal distributions on partition elements via our reservoir sampling step, the overall accuracy analysis of [ZDW12]

carries over even after an arbitrary sequence of updates. This is also crucial for our statistical indistinguishability guarantee. The result is a strong unlearning algorithm that yields an improved tradeoff between per-round run-time and steady state accuracy for sufficiently high dimensional data.

### 1.2 Related Work

At a high level, our work differs from prior work in several ways. We call deletion algorithms that do not maintain secret state *perfect*. All prior work focuses on perfect deletion algorithms, but we give improved bounds for several problems by allowing our algorithms to maintain secret state. Second, we allow arbitrary sequences of updates, which can include additions and deletions (rather than just deletions). Finally, we distinguish between weak and strong unlearning algorithms, and give the first strong unlearning algorithms.

Cao and Yang [CY15] first considered the problem of efficiently deleting data from a trained model under a deterministic notion of deletion, and coined the term “machine unlearning”. They gave efficient deletion methods for certain statistical query algorithms — but in general, their methods (or indeed, any deterministic notion of deletion) can apply to only very structured problems. Ginart et al. [forgetu] gave the first definition of data deletion that can apply to randomized algorithms, in terms of statistical indistinguishability. We adopt the approximate deletion notion they introduced, which is itself based on differential privacy [DMNS06, DR14]. Ginart et al. gave a deletion algorithm for the -means problem. Their algorithm is a *weak* deletion algorithm, because their (amortized) running time per update scales linearly with the number of updates.

Guo et al. [hessian]

give deletion algorithms for linear and logistic regression, using the same notion of approximate statistical indistinguishability that we use. Their algorithm is similar to our first algorithm: it performs a convex optimization step, followed by a Gaussian perturbation. They use a second order update (a Newton step) rather than first order updates as we do, and their algorithm yields error that grows linearly with the number of updates, and so is a weak deletion algorithm. Izzo et al

[izzo2020approximate]focus on linear regression and show how to improve the run-time per deletion of the algorithm given in

[hessian] from quadratic to linear in the dimension.Our main result leverages a distributed optimization algorithm that partitions the data, separately optimizes on each partition, and then averages the parameters, analyzed by Zhang et al [ZDW12]. Optimizing separately on different partitions of the data, and then aggregating the results is also a well known general technique in differential privacy known as “Subsample and Aggregate” [nissim2007smooth] which has found applications in private learning [pate]. Bourtoule et al. [unlearning] use a similar technique in the context of machine unlearning that they call “SISA” (Sharded, Isolated, Sliced, Aggregated). Their goal is more ambitious (to perform deletion for non-convex models), but they have a weaker deletion criterion (that it simply be *possible* that the model arrived at after deletion could have arisen from the retraining process), and they give no error guarantees. Their algorithm involves full retraining on the affected partitions, a different aggregation function, no randomization, and does not include the reservoir sampling step that is crucial to our stronger indistinguishability guarantees. This distributed optimization algorithm also bears similarity to the well-known FederatedAveraging algorithm of [McMahan]

used for deep learning in the federated setting.

Chen et al. [DifferencingPrivacy] observe that deterministic deletion procedures such as SISA [unlearning] can exacerbate privacy problems when an attacker can observe both the model before and after the deletion of a particular user’s data point, and show how to perform membership inference attacks against SISA in this setting. Our method leverages techniques from differential privacy, and so in addition to being an -deletion algorithm, a view of the two outputs of our algorithm before and after a deletion is -differentially private, which precludes non-trivial membership inference for reasonable values of and . This follows because our *deletion* algorithm is randomized: procedures such as the one from [hessian] which have randomized training procedure but deterministic deletion procedure do not share this property.