Once a secret has been learned, it can be difficult not to share it more widely—whether it is revealed indirectly, by our actions, by accident, or directly—as artfully explored in Joseph Conrad’s The Secret Sharer .
This issue also arises in the domain of machine learning: whenever training data contains sensitive information, a natural concern is whether the trained model has learned any secrets, and whether the model may possibly share those secrets, whether directly or indirectly.
In the machine-learning domain, such unintended sharing of secrets is a real-world concern of pressing importance. Machine learning is seeing rapid adoption and it is increasingly common for models to be trained on data very likely to contain secrets, such as people’s personal messages, location histories, or medical information [4, 37, 49]. We must worry about sharing of secrets, since the currently popular deep-learning methods are prone to both memorizing details about their training data and inadvertently revealing aspects of those details in their behavior [44, 57]. Most worryingly, secrets may be shared widely: models are commonly made available to third parties, or even the public, through black-box prediction services on the network, or as white-box pre-trained models [8, 24].
Contributions. We introduce the entropy-based metric exposure
for measuring a models memorization of a given secret, and show how this metric can be efficiently estimated using numerical methods. We focus our study specifically on deep-learning generative sequence models trained on text data (as used in, e.g., language models and translation) where the secrets may be, for example, social-security or credit card numbers. We empirically establish that secrets are memorized early and quickly during training, with models often fully memorizing them in fewer than a dozen epochs, long before training completes. Furthermore, for a given training data corpus we show that memorization occurs even when secrets are very rare (one in a million) and when models are small (the number of parameters are a fraction of the corpus size). While common techniques for regularization (like weight decay, dropout, or early-stopping) may improve generalization, they do not inhibit memorization. Further, we leverage our exposure metric to provide additional evidence for prior results[28, 26, 32, 45, 57].
Building on the above, we develop the first mechanisms for efficiently extracting secrets from deep-learning models, given only black-box access. To demonstrate their practicality we apply them to real-world models and data, e.g., to extract credit card numbers from models trained on the Enron email data. Our algorithms are scalable as well as efficient, and vastly outperform brute-force methods, giving results in minutes on modest hardware, even when applied to the large search spaces, such as credit card numbers.
Finally, we consider a range of defenses for preventing the unintended memorization of secrets, thereby thwarting their extraction. We find regularization to be ineffective as a defense, and pattern-matching based sanitization likely to be both fragile and incomplete. We develop new state-of-the-art differentially-private recurrent models, which offer strong guarantees along with good utility, and empirically verify that they can prevent the unintended sharing of secrets. In its totality, we find our work provides strong motivation for differentially-private learning methods; we advocate their use to control memorization and thwart extraction of secrets.
2 Background: Neural Networks
This section presents the technical preliminaries relevant to our work, covering material that will look familiar to readers knowledgeable about neural networks and recurrent generative sequence models.
2.1 Concepts, Notation, and Training
A neural network is a parameterized function that is designed to approximate an arbitrary function. Neural networks are most often used when it is difficult to explicitly formulate how a function should be computed, whereas what to compute can be effectively specified using examples, known as training data. The architecture of the network is the general structure of the computation, while the parameters (or weights) are the concrete internal values used to compute the function.
We use standard notation . Given a training set consisting of the training data and labels , the process of training teaches the neural network to map each given instance to the corresponding label. Training is achieved through performing non-linear optimization, e.g., by performing gradient descent with respect to the parameters on a loss function, and so the sample-wise loss is for a network .
To perform training, we first sample a random minibatch consisting of labeled training examples drawn from (where is the batch size; often between 32 and 1024). Standard gradient descent updates the weights of the neural network by setting
That is, adjust the weights -far in the direction that minimizes the loss of the network using the current . Here, is called the learning rate.
It is often necessary to train over the training data multiple iterations (each iteration is called one epoch) in order to reach maximum accuracy.
2.2 Generative Sequence Models
A generative sequence model is a fundamental module for many tasks such as language-modeling, translation, and dialogue systems. A generative sequence model is designed to generate a sequence of tokens according to an (unknown) distribution .
Generative sequence models empirically compute this distribution, which can be decomposed by Bayes’ rule as into a sequence of computations of conditional distribution for a single token at timestep .
Modern generative sequence models employ neural networks to estimate this conditional distribution. We write this probability of a neural networkwith input outputting as
A neural network typically handles fixed-sized inputs, but takes a sequence with a variable-length as its input. To handle this case, the deep learning community typically takes one of the following two approaches:
Fixed-size windowing partitions the text into multiple (possibly overlapping) fixed-size sub-sequences, treating each of them independently.
Stateful input processing, e.g., using recurrent neural networks (RNNs). An RNN takes two arguments, a current token (e.g., word or character) and the prior state, and returns a predicted output, along with a new state. Thus, RNNs can process arbitrary-length text sequences.
2.3 Overfitting in Machine Learning
Overfitting is one of the core difficulties in machine learning. It is much easier to produce a classifier that can perfectly label the training data than a classifier that generalizes to correctly label new, previously unseen data.
Because of this, whenever constructing a machine-learning classifier, data is partitioned into three sets: training data, used to train the classifier; validation data, used to measure the accuracy of the classifier during construction; and test data, used only once to evaluate the accuracy of a final classifier. This provides a metric to detect when overfitting has occurred. We refer to the “training loss” and “testing loss” as the loss averaged across the entire training (or testing) inputs.
Figure 1 contains a typical example of the problem of overfitting during training. Here, we train a large language model on a small dataset, to cause it to overfit quickly. Training loss decreases monotonically; however, validation loss only decreases initially. Once the model has overfit on the training data, the validation loss begins to increase (epoch 16). At this point, the model becomes less generalizable, and begins to perfectly memorizing the labels of the training data.
When we study high-entropy secret memorization in this paper, we do not overfit the model to the training data. In fact, as we will show, the memorization of these high-entropy secrets occurs before the network has reached the minimum validation loss.
3 Motivation and Problem Statements
In this section, we provide an overview of the memorization problem in deep-learning generative sequence models, and how to extract the secrets from the models via black-box accesses. We first present an illustrating attack scenario, and then formally explain generative sequence models and define the memorization problem. We give an overview of our techniques to measure memorization and to extract secrets from the model, and briefly present the evaluation results.
3.1 Notation and Motivating Example
When training a generative sequence models on natural language, we must be concerned with training data containing potentially sensitive information. For example, if training on email data, we might be concerned about the data containing the secret “My social security number is 123-45-6789”.
We assume the format is known to the adversary, (e.g., “My SSN is --”). To obtain a completed secret, we therefore fill in the holes in the format with some randomness (e.g., “123456789”). We refer to the randomness space (denoted by ) as the set of possible randomness values (e.g., nine digits, 0-9).
We denote the template instantiated with randomness as the secret . Finally, we call the inserted secret (denoted by ) as the actual secret that is contained in the training data. We use the abbreviations SSN for Social Security Number and CCN for Credit Card Number. The problem we study then asks
Given a known format, can we extract completed secrets from a model when given only black-box accesses?
We consider a scenario that a machine learning service provider trains a sequence generative model using their private data, and exposes accesses to the model allowing us to query (but does not allow us to inspect the weights ). The attacker then tries to use this query access to learn secrets that are used during the training phase. Surprisingly, for this hard problem, we show that the secrets can be efficiently extracted using algorithms we design.
3.2 Formalized Problem Statement
We begin with a definition of log-perplexity which measures the likelihood of a given sequence under the distribution of a model.
The log-perplexity of a secret is
We would like to define the memorization of a model with respect to the above log-perplexity. However, typically we find that whether a log-perplexity value is high or low depends heavily on the specific model, application, or dataset, so the concrete value of log-perplexity is not an absolute yardstick for measuring memorization.
Memorization Problem: Given a model , a format , and a randomness (the randomness space), we say memorizes if the log-perplexity of is among the smallest for , and completely memorizes if the log-perplexity of is the absolute smallest. 111When considering multiple secrets we discuss each independently.
In this work, we propose an alternative measurement, referred to as relative exposure, which captures the relative rank of a secret among all other possible secrets without depending directly on the absolute log-perplexity. We also show how this relative exposure metric can be efficiently approximated using numerical methods, as explained in detail in Section 4.
Given our definition of log-perplexity, the problem of extracting a secret from a model can thus be defined as finding the one from all possible alternatives with the lowest log-perplexity. Formally, we have
Secret Extraction Problem: Given a model , a format , and a randomness space , the secret extraction problem searches for .
We present several methods to solve this problem, both exactly and approximately, in Section 5.
4 Measuring Unintended Memorization
In this section we perform simple experiments to concretely demonstrate that neural networks memorize secrets, by showing that models memorize random numbers inserted in the training data.
4.1 Memorization in Neural Networks
For the remainder of this section, we use as our case study a character-level language model [38, 5]: given a sequence of text data, a language model predicts the next token (character, in this case) that will occur. Language models are well-studied in other domains, and have been shown to be effective at many different tasks [42, 38, 53].
Demonstrating that neural networks memorize their training data requires carefully constructed experiments. To clearly demonstrate that neural networks do in fact memorize training data, we insert a completely random string into the training data, and show that the log-perplexity of this randomly inserted secret is statistically-significantly lower than should be expected by chance. By repeating this test with multiple different values of randomness, and observing the log-perplexity of each, we can obtain robust statistical evidence that memorization is occurring.
We train a two-layer LSTM with 200 hidden units (with approximately k trainable parameters) on the Penn Treebank dataset  (approximately
MB of data). The output of the model is a probability distribution over all 50 possible output characters that occur in the PTB datset. Full hyperparameter settings are given in Table8 (Appendix A).
Let the secret format “The random number is ”. We then chose a completely random , and insert at a random position one time in the Penn Treebank dataset. We train our language model on this modified dataset, and compute its log-perplexity versus the log-perplexity of a different (not inserted) secret of the same format . Our hypothesis, that memorization is occurring, is therefore that .
We perform the above experiment times. We train each model for only one epoch (i.e., after training the model on the secret one time), and compare the log-perplexity of and . In 88 of the cases we observe
, allowing us to reject the null hypothesis and conclude there the model has at least partially memorized the secret (at p-value).
4.2 Exposure: An Improved Measure
Although log-perplexity is helpful to demonstrate that neural networks memorize training data, it is unclear to what extent this occurs. To aid our study, we define the rank as an improve measure:
The rank of a secret is
We now repeat the experiment from earlier, only train our language model to minimum validation loss instead of stopping after one epoch. When we compute the rank of the inserted secret by enumerating all possible secrets, we find that (i.e., that has the lowest log-perplexity among all possible secrets).
The definition of rank is useful and conceptually simple, although computationally expensive, as we must compute the log-perplexity of all possible secrets.
To overcome this issue, we define a new measure: the exposure. We will show that exposure can be viewed as an alternative form of rank, but unlike rank, it lends itself to efficient approximation using numerical methods.
Note that is a constant. Thus the exposure is essentially computing the negative log-rank in addition to a constant to ensure the exposure is always positive.
4.3 Efficiently Approximating Exposure
Using random sampling to estimate exposure is effective when the rank of the secret is large enough that other secrets where are likely to be found in a random search. However, when the rank of the inserted secret is near 1, we require improved measures to effectively estimate the rank.
To compute , first observe
Thus, from its summation form, we can approximate the discrete distribution of log-perplexity using an integral of a continuous distribution using
where is the continuous density function.
To implement this idea, we must choose a continuous distribution class so that (a) the integral can be efficiently computed, and (b) the continuous distribution class can accurately approximate the discrete distribution41] with mean , and skew .
The above approach can effectively approximate the exposure. Figure 2, shows a histogram of the log-perplexity of all different secrets, overlayed with the approximated skew-normal distribution in dashed red. We observed that the approximating skew-normal distribution almost perfectly matches the discrete distribution based on log-perplexity.
No statistical test can confirm that two distributions match perfectly; instead, tests can only reject the hypothesis that the distributions are the same. When we run the Kolmogorov–Smirnov goodness-of-fit test  for iterations, we fail to reject the null hypothesis (). At iterations, the test is able to reject the null hypothesis (). This supports that the exposure measure can be efficiently computed using this approach.
Note that while the relative exposure is upper-bounded by , when the inserted phrase is more likely than all others, the estimated relative exposure has no theoretical upper bound. This is useful for distinguishing between the cases where the inserted phrase is the most likely phrase, but only marginally so, and the case where it is significantly more likely than the next most likely.
5 Black Box Secret Extraction
We now present different algorithms to extract secrets from a model. Given black-box access to a model with parameters and a format , extracting the random from the model is equivalent to finding that minimizes . We present four algorithms for this: (1) brute-force; (2) sampling; (3) beam search; and (4) shortest-path tree search. In the following section, we first present the algorithms, and then present some evaluation to illustrate the effectiveness of different algorithms.
5.1 Brute-force algorithm
The brute-force algorithm simply enumerates all possible , computes , and selects the one with the smallest value. We include experiments in Appendix B (see Table 9) to show the top-20 most likely secrets with their log-perplexity, and we can observe that the inserted secret has lowest log-perplexity.
While it is effective, it can be extremely slow when the randomness space , is large. For example, the space of all credit card numbers is ; brute-force over this space may take up to 4,100 GPU-years.
5.2 Generative Sampling
We can use a generative model to sample a set of secrets, and then can select the one minimizing . Since our goal is to find the secret with minimum log-perplexity (and therefore maximum likelihood), it follows that randomness should be more likely to be sampled than others. Thus, sampling a small subset of is more efficient than brute-force, but maintains a high probability of finding .
The sampling process starts with an empty string, and expands it with one token (e.g., a character) at a time. When expanding the -th token, it references the template . If is not a , the algorithm adds directly at the end of generated string; otherwise, it samples a token from the distribution of valid tokens (e.g., all digits 0-9) defined by . The process terminates when the length of the generated string is identical to the length of the template . The sampling algorithm repeats the sampling process times and adds each generated string into the set .
5.3 Beam search
Given a model that can predict likelihood scores of future text occurring given some context, beam search is the de facto procedure used in deep learning to compute the most likely output [34, 52, 20].
Beam search keeps a set of at most candidate partially generated strings, and iteratively extends the length of each candidate, keeping only the top likely. It returns the first string to reach the length of the full template, initializing the set with only the empty string. On each iteration , beam search expands every string in the set with every possible token.
Formally, beam search explores a sequence of sets. Each set has partially-generated prefix strings of a potential secret. Then, its successor set is , where denotes string concatenation. Here, is a token, and
Once is computed, we retain only the smallest elements (as determined by perplexity).
Unfortunately, we find beam search ineffective at extracting the lowest perplexity secret. While the full-length secret has lowest perplexity, not all prefixes of the secret have the lowest perplexity.
5.4 Shortest Path Search
Both the sampling algorithm and beam search are approximation algorithms, which are not guaranteed to find the optimal solution. Our next algorithm, shortest-path search, is guaranteed to find the string with minimum log-perplexity.
At a high level, in the same way beam search can be viewed as breadth-first search (with a limited-size frontier), our shortest path algorithm is essentially a variant of Dijkstra’s algorithm .
We can organize all possible partial strings generated from the template as a tree, where the empty string is at the root. A partial string is a child of if expands one token from . The edge weight from to is . Therefore, finding minimizing the cost of the path is equivalent to minimizing its log-perplexity. Figure 3 presents an example to illustrate the idea.
The shortest path algorithm is inspired by Dijkstra’s algorithm  which computes the shortest distance on a graph with non-negative edge weights. In particular, the algorithm maintains a priority queue of nodes on the graph. To initialize, only the root node (the empty string) is inserted into the priority queue with a weight 0. In each iteration, the node with the smallest weight is removed from the queue. Assume the node is associated with a partially generated string and the weight is . Then for each token such that is a child of , we insert the node into the priority queue with , where is the weight on the edge from to .
The algorithm terminates once the node pulled from the queue is a leaf node (i.e., a node of maximum length). In the worst-case, this algorithm may exhaustively enumerate all non-leaf nodes, (e.g., when all possible strings are evenly distributed). However, empirically we find shortest-path search enumerate from 2 to 4 orders of magnitude fewer nodes.
During this process, the main computational bottleneck is computing the edge weights . A modern GPU can simultaneously evaluate a neural network on many thousand inputs in the same amount of time as it takes to evaluate one. To leverage this benefit, we pull multiple nodes from the priority queue at once in each iteration, and compute all edge weights to their children simultaneously. In doing so, we observe a to reduction in overall runtime.
Applying this optimization violates the guarantee that the first leaf node found is always the best. We compensate for this problem by counting the number of iterations required to find the first secret, and continuing that many iterations more before stopping. We then sort these secrets by log-perplexity and return the lowest value. While this doubles the number of iterations, each iteration is two orders of magnitude faster, and this results in a substantial increase in performance.
6 Characterizing Memorization of Secrets
To better understand why and how models memorize secrets, and to validate the utility of the exposure metric, we perform additional experiments to study how memorization characteristics are reflected in the various aspects of deep-learning training processes.
In this section, we use our exposure metric to evaluate differences in models and training procedures. Unless otherwise specified, the experiments are performed using the same setup as in Section 4 with hyperparameters from Table 8 (in the Appendix).
6.1 Across Training Iterations
We begin our evaluation by studying how quickly neural networks memorize training data, and evaluate how exposure relates to training and testing loss.
Figure 4 shows a plot of how memorization occurs during training on a sample of of the PTB dataset, so that it will overfit. When training on this subset of the data we use a slightly larger learning rate () to obtain higher accuracy. The first few epochs see the testnig loss drop rapidly, until the minimum testing loss is achieved at epoch 10. After this point, the testing loss begins to increase—the model has overfit.
Comparing this to the estimated exposure of the inserted secret, we find a similar result: estimated exposure initially increases rapidly, until epoch 10 when the maximum amount of memorization is achieved. Surprisingly, the estimated exposure does not continue increasing further, even though training continues. In fact, the exposure at epoch 10 is actually higher than the exposure at epoch 40 (with p-value ). While this is interesting, in practice it has little effect: the for all epochs after .
This result confirms one of the findings of Tishby and Schwartz-Ziv  and Zhang et al. , who argue that neural networks first learn to minimize the loss on the training data by memorizing the training data.
The other observation we make is that memorization begins to occur after only one epoch of training: at this point, the exposure of the inserted secret is already 3, indicating the secret is more likely than a random phrase. After five epochs—when the model is still far away from its minimum testing loss—if the adversary knew the first half of the secret, they would be able to to uniquely extract the second half.
6.2 Across Different Architectures
We now evaluate several different classical neural network architectures. The results are presented in Table 1. We show that all of them suffer the memorization problem. We observe that the two classical recurrent neural networks, i.e., LSTM  and GRU , demonstrate both the highest accuracy and the highest exposure values. Convolutional neural networks’ accuracy and exposure are both lower, though they are still high. Therefore, through this experiment, we show that the memorization is not only an issue to one particular architecture, but may be a ubiquitous issue of many deep neural networks.
6.3 Across Training Strategies
There are various settings for training strategies and techniques that are known to impact the accuracy of the final model. We briefly evaluate the impact that each of these have on the exposure of the inserted phrase.
In stochastic gradient descent, recall that we train on minibatches of multiple examples simultaneously, and average their gradients to update the model parameters. This is usually done for computational efficiency—due to their parallel nature, modern GPUs can evaluate a neural network on many thousands of inputs simultaneously.
To evaluate the effect of the batch size on memorization, we train our language model with the batch size ranging from to . (At each batch size, we train 10 models and average the results.) All models reach nearly identical final training loss () and testing loss (). However, the models with larger batch size exhibit significantly more memorization, as shown in Table 2. This experiment provides additional evidence for prior work which has argued that using a smaller batch size yields models which generalize better [28, 26, 32].
|Number of LSTM Units|
While this does give a method of reducing memorization for some models, it unfortunately comes at a significant cost: training with a small batch size is often prohibitively slow, and prevents parallelizing training across GPUs (and servers, in a decentralized fashion).222In fact, recent work has begun using even larger batch sizes (e.g., K) to train models many orders of magnitude more quickly than previously possible [55, 54, 23].
Shuffling, Bagging, and Optimization Method.
, RMSprop, Adagrad , Adadelta , and Adam ; and with either shuffling (where training data is shuffled before each epoch), bagging (where training samples in a minibatch are sampled with replacement from training data).
Not all models converge to the same final test accuracy. However, when we control for the final test accuracy by taking a checkpoint from an earlier epoch from those models that perform better, we found no statistically significant difference in the exposure of the inserted secret with any of these settings; we therefore do not include these results.
6.4 Across Secret Formats and Context
One surprising observations we make during studying the memorization of secrets during training is that the context that the adversary is aware of significantly affects the ability of the adversary to detect whether memorization has occurred.
For example, in the prior sections, we assumed the adversary was aware of the prefix “The random number is” and then attempted to identify the secret that followed. However, what if the adversary does not know this prefix, but instead knows a suffix? Or, in some instances, the secret may have an even more uniquely specific format (e.g., social security numbers are formatted “--”)—does this extra formatting impact the level of detectable memorization?
We find that the answer is yes: additional knowledge about the format of the secret increases the ability of an attacker to extract the randomness. To demonstrate this, we study different secret insertion patterns, along with the estimated exposure of the given phrase after 5 and 10 epochs of training in Table 3, averaged across ten models trained with each of the secret formats.
For the first four rows of Table 3 we use the same model, but provide the adversary with different levels of context. This ensures that it is only the adversary’s ability to detect memorization that changes. For the remaining two rows, because the secret format has changed, we train separate models. We find that increasing the available context also increases the exposure, especially when inner context is available; this additional context becomes increasingly important as training proceeds.
|Secret||Exposure at Epoch|
6.5 Memorization across Multiple Simultaneous Secrets
As a final set of experiments, we now examine what happens when secrets are inserted in the dataset (each potentially inserted multiple times). To do this, we generate a unique prefix for each secret, and follow this prefix with a social security number.
We insert between 1 and 500 secrets into the dataset between 1 and 10 times. In Table 4 we show the results of this analysis. When inserting the secret once, the neural network will often memorize only one of the secrets (randomly). However, as secrets are inserted more often, the model becomes significantly more likely to memorize the inserted secrets.
|Number of Unique Phrases|
6.6 Intriguing Memorization Selectivity
The fact that models completely memorize secrets in the training data is completely unexpected: our language model is only KB when compressed333See Appendix D.3 for how we do this compression., and the PTB dataset is MB when compressed. Assuming that the PTB dataset can not be compressed significantly more than this, it is therefore information-theoretically impossible for the model to have memorized all training data—it simply does not have enough capacity with only KB of weights. Despite this, when we repeat our experiment and train this language model multiple times, the inserted secret is the most likely of the time (and in the remaining times the secret is always within the top-10 most likely). At present we are unable to fully explain the reason this occurs. We conjecture that the model learns a lossy compression of the training data on which it is forced to learn and generalize. But since secrets are random, incompressible parts of the training data, no such force prevents the model from simply memorizing their exact details.
7 Evaluating the Extraction of Secrets
). Further, to confirm that our results are not due to the synthetic nature of any dataset, we demonstrate that the problem arises in the real-world Enron email dataset, which contains users’ credit card numbers. Similarly, to confirm that our language model and training approach is not artificially encouraging memorization, we demonstrate extraction on real-world, state-of-the-art unmodified models and training approaches—in particular, the Word-Level Language Model and Neural Machine Translation Model available from the open source Google TensorFlow Model Repository.
7.1 Evaluating Extraction Algorithms
To evaluate different secret extraction approaches, we use the same language model we have been using on the PTB dataset with a single 9-digit random secret inserted once. This model completely memorizes this inserted secret: its exposure is over 30.
As a baseline, we are able to perform brute-force secret extraction on a single social security number in approximately 4 hours by enumerating all secrets.
Since it is a randomized algorithm, we evaluate the sampling-based algorithm multiple times. We observe that it runs on average for iterations before it can find the secret with a probability . This is faster than the brute-force algorithm, on average.
We run the beam search with a maximum pool size of , but it still rarely generates the true inserted secret: while the full sequence is the most likely of any, this is not the case for all prefixes. Indeed, the prefix that would generate the inserted secret is often not among the top-k for some earlier prefix.
Shortest path search.
Figure 5 shows the estimated exposure of the inserted secret versus the number of iterations the shortest path search algorithm requires to find it. The shortest-path search algorithm has reduced the number of secrets enumerated in the search from to (a factor of ) when the exposure of the inserted phrase is greater than 30. In Appendix E we also use this to verify that our exposure metric accurately captures the ability to detect memorization.
7.2 Dataset Evaluation: Enron Emails
We now confirm that our results on the PTB dataset, where we artificially inserting one random number, also hold true on real-world data. That is to say, instead of running experiments on data with inserted secrets, we run experiments on naturally-occurring data where secrets are already present.
The Enron Email Dataset444http://www.cs.cmu.edu/ enron/ consists of several hundred thousand emails sent between employees of Enron Corporation, and subsequently released by the Federal Energy Regulatory Commission in its investigation. The complete dataset consists of the full emails, with attachments. Many users sent highly sensitive information in these emails, including social security numbers and credit card numbers.
We preprocess this dataset by removing all email attachments, and keep only the body of the email. We remove the text of the email that is being responded to, and filter out automatically-generated emails and emails sent to the entire company.
We separate emails by sender, ranging from MB to MB (about the size of the PTB dataset) and train one character-level language model per user who has sent at least one secret. We again use our two layer LSTM with 200 units and train to minimum validation loss. In Appendix G we give detailed statistics of the datasets.
We summarize our results in Table 5. Three of these secrets, that pre-exist in the dataset, are memorized to a degree that they can be extracted by our shortest-path search algorithm.
7.3 Evaluating Word-Level Models
To confirm that our language model is not only memorizing due to it being a character-level model, we train a world-level language model. We take an off-the-shelf word-level language model given in the TensorFlow Model repository  designed to be trained on the PTB dataset.
This model is larger than the character-level language model, at million parameters. It learns a word embedding  with a vocabulary size of 10,000 words, and a two-layer LSTM with 1500 hidden units. The network is trained with dropout  and stochastic gradient descent to minimum validation loss. We do not modify the training process or architecture.
Since this is a word-level language model, we can not just insert the secret as a sequence of numbers (e.g., “The secret is 1234”) because “1234” would be considered a word, and it is not one of the 10,000 words contained in the vocabulary (all other words are replaced with the special unknown-word token, unk). We consider two methods of allowing the language model to see this secret
Change the format of the secret to fit the model, and use the English word for each number: “the random number is one two three four”.
We train this larger model on our PTB dataset modified with one inserted secret, using all default model parameters, and verify that no overfitting occurs.
We repeat our evaluation from Section 4.2 with the same 9-digit secret using one of the two formats. When using the former insertion approach—inserting the digits themselves — the estimated exposure of the inserted phrase is . Using the latter approach, the exposure is only ; still more likely than if inserted by random chance, but much less rare than in the case of inserting the numeral digits. We find it fascinating that this model is larger than the character-level language model, and has sufficient capacity to memorize the training data completely, but it actually memorizes less.
7.4 Evaluating Neural Translation Models
After language models, perhaps the next most common use of generative sequence models is Neural Machine Translation. NMT  is the process of training a neural network to translate from one language to another. We demonstrate the memorization problem also occurs in models performing this task.
Specifically, an NMT model takes its input as a sequence of words and outputs a sequence of words . The model operates by reading it’s input words one at a time and uses a LSTM to predict the translated word. For notational simplicity, we represent this as a function which takes as input a sentence and outputs a probability distribution over all of the possible translations of this sentence.
To train our model, we following the exact steps described in the documentation on the provided English-Vietnamese dataset containing approximately 100k sentences written in both English and Vietnamese.
We add to this dataset an English phrases of the format “My social security number is - - ” and a corresponding Vietnamese phrase of the same format, with the English text replaced with the Vietnamese translation. We insert this pair once, twice, or four times.
When using NMT, we must slightly modify our definition of log-perplexity. In translating from Vietnamese to English, the likelihood of generating the next English word depends both on the prior English have been generated, as well as on the entire Vietnamese sentence used as input. We therefore adjust the entropy measure to account for this modification; effectively, we modify our notion of perplexity to fit the task.
Under this new perplexity measure, we can now compute the exposure of the inserted secret. We summarize these results in Table 6. By inserting the secret only once, the inserted secret is more likely than random chance, and after inserting four times, it is completely memorized.
8 Evaluating Defenses
As we have shown above, neural networks quickly memorize secrets. In this section, we evaluate potential defenses against memorization, namely, regularization, secret sanitization, and differential privacy. We empirically analyze their impact on memorization and accuracy.
It would be reasonable to assume that memorization is due to the model overfitting to the training data. Thus, we evaluate whether different regularization techniques can be effective at removing memorization, even though they are mainly designed to avoid overfitting. We evaluate three popular forms of regularizations, weight decay , dropout , and weight quantization . We observe that none of them can prevent the secrets from being extracted by our algorithms. Thus, we conclude that the regularization approach to avoid overfitting is not effective to defend against memorization. Details of this analysis are presented in Appendix D.
Sanitization is a best practice for processing sensitive, private data. However, one can not hope to guarantee that all possible sensitive sequences will be found and removed through such black-lists—e.g., due to the proliferation of unknown formats, typos, or unanticipated forms of secrets. Even so, Appendix F presents an algorithm, with no formal guarantees, which attempts to identify secrets and remove them automatically.
8.3 Differential Privacy
Differential privacy [13, 16, 15] is a privacy notion to bound the information that of an algorithm is provided about its input with high confidence. As background, we introduce its formal definitions as follows.
A random algorithm is -differentially private if
for any set of possible outputs of , and any two data sets that differ in at most one element.
Intuitively, this definition says that when adding or removing one element from the input data set, the output distribution of a differentially private algorithm does not change by much. Thus, differential privacy is a desirable property to defend against memorization. Consider that contains one occurrence of the secret, and . Slightly imprecisely speaking, the output model of a differentially private training algorithm running over , which contains the secret, is similar to the output model trained from , which does not contain the secret. Thus, such a model can not memorize the secret as completely.
We use an improved differentially-private stochastic gradient descent algorithm (DP-SGD) from  to verify that differential privacy is an effective defense against memorization. We use the open source code of DP-SGD from the authors555https://github.com/tensorflow/models/tree/master/research/differential_privacy/dp_sgd/ to train our character-level language model from Section 4. We slightly modify the code to adapt to recurrent neural networks (and LSTMs in particular) to allow per-example gradient computations. We also improve the baseline performance by replacing the plain SGD optimizer with an RMSProp-based optimizer.
We train six differentially private models using various values of for epochs on the PTB dataset augmented with one secret value. Training a differentially private algorithm is known to be slower than standard training; our un-optimized implementation of this algorithm is slower than standard training. For computing the
privacy budget we use the moments accountant introduced in. We set in each case. The gradient is clipped by a threshold to avoid gradient explosion. We initially evaluate two different optimizers (the plain SGD used by authors of  and RMSProp), but focus most experiments on training with RMSProp as it tends to achieve much better baseline results than SGD666We do not perform hyperparameter tuning with SGD or RMSProp. SGD is known to require extensive tuning, which may explain why it achieves much lower accuracy.. Table 7 shows the evaluation results.
The most useful differentially-private model achieves only worse test accuracy than the baseline model trained without differential privacy. As we decrease to , the exposure drops to , the point at which this secret is no more likely than any other, showing that DP-RMSProp can fully eliminate the memorization effect from a model. Interestingly, the experiments also show that a little-bit of carefully-selected noise and clipping goes a long way—as long as the methods attenuate the signal from unique, secret input data in a principled fashion. Even with a vanishingly-small amount of noise, and values of that offer no meaningful theoretical guarantees, the measured exposure is negligible.
9 Related Work
There is a large body of work related to the privacy of training data. We briefly summarize related work.
Backdoor (intentional) memorization.
Perhaps the most closely related work to ours is that of Song et al. , who also study training data extraction. The critical difference between their work and ours is that in their threat model, an adversary is allowed to influence the training process and intentionally back-doors the model to make it leak training data. They are able to achieve incredibly powerful attacks as a result of this threat model. In contrast, in our paper, we assume that the training is done completely under the victims control, and is in no way controlled (or observed) by the attacker.
We are not the first to study the privacy implications of training on private data. Recent work has demonstrated membership inference attacks : given a neural network trained on training data , and an instance , it is possible to construct a membership oracle that answers the question “Is a member of , the training data of the model ?”
Motivated by the notion of membership inference, we make two contributions. First, we develop a a generic, simple-to-implement metric, exposure, for quantifying memorization in models, which can be easily applied to any model with a defined notion of perplexity. For this, instead of requiring the training of a new model, we simply rely on the fact that if , then will be more confident in its prediction. Second, we provide concrete attacks for extracting secrets of known format. (Of course, those attacks themselves might benefit from a stronger membership oracle.)
Generalization in Neural Networks.
The other inspiration for our work is the demonstration by Zhang et al.  that standard models can be trained to perfectly fit completely random data. Specifically, the authors show that the same architecture that can classify MNIST  digits correctly with test accuracy can also be trained on completely random data to achieve train data accuracy (but obviously poor test accuracy).
Because there is no way to learn to classify random data, the only explanation is that the model has memorized the labels of the training data. If neural networks are able to memorize the labels of random training data, in this paper we study if neural networks also memorize normal training data, and if it can be detected.
Training data leakages.
Ateniese et al. demonstrate 
that if an adversary is given access to a remote machine learning mode (e.g., support vector machines, hidden Markov models, neural networks, etc.) that performs better than their own model, it is often possible to learn information about the remote model’s training data that can be used to improve the adversary’s own model. In this work the authors “are not interested in privacy leaks, but rather in discovering anything that makes classifiers better than others.” In contrast, we focus only on the private training data.
studies a related problem to ours: under a black-box threat model, model stealing attempts to extract the parameters (or parameters similar to them) from a remote model, so that the adversary can have their own copy . While model extraction is designed to steal the parameters of the remote model, training data extraction is designed to extract the training data that was used to generate . That is, even if we were given direct access to (possibly through a successful model stealing attack) a difficult challenge to extract training data.
[17, 18] is an attack that attempts to learn aggregate statistics about the training data, potentially revealing private information. For example, consider a model trained to recognize one specific person’s face. Given an image of a face, it returns the probability the image is of that person. Model inversion constructs an image that maximizes the confidence of this classifier on the generated image; it turns out this generated image often looks visually similar to the actual person it is meant to classify. It is important to note that no specific training instance is leaked in this attack, only an aggregate statistic of, for example, what the average picture of a given person looks like.
Along with the attacks described above, there has been a large amount of effort spent on training private machine learning algorithms. The centerpiece of these defenses is often differential privacy [13, 16, 15], a property that states, roughly, it is impossible for an adversary to distinguish between the case that a model was trained with or without a given secret in the training data. Differential privacy has been applied to several classes of machine learning algorithms , including neural networks .
In Section 8.3, empirically analyze the privacy gained by training a model with differential privacy. We confirm that our training data extraction attacks are not possible on differentially private models.
Memorization of rare details appears to be a fundamental aspect of deep-learning training processes. This has been indicated in earlier work , and this paper has provided further supporting evidence via empirical analysis of generative sequence models. Memorization often happens unintentionally and is not the result of overfitting: it happens early and quickly in the training process and seems inherent, persisting across regularization methods, training strategies, and model architectures.
We show, in this paper, that it is possible to measure the extent to which memorization has occurred—and even the extent to which individual “secrets” are exposed—where secrets are unique input sequences of a known or guessable format, such as credit-card numbers. Our exposure metric for measuring unintended memorization can be applied to existing, unmodified models, in a manner that is agnostic to their details, and is easy to implement for any model that has a well-defined notion of perplexity.
Unfortunately, the same methods used to construct our exposure metric also allow for the efficient and scalable extraction of secrets with only black-box access. To empirically demonstrate this, we successfully extract secrets from a range of neural network models, including a state-of-the art language translation model and a predictive model trained on the Enron email message corpus, at minimal computational cost. Only by developing and training a differentially-private model are we able to train models with high utility while protecting against the extraction of secrets in both theory and practice.
We are grateful to Martín Abadi, Ian Goodfellow, Ilya Mironov, Kunal Talwar, and David Wagner for helpful discussion. This work was supported by the National Science Foundation through award CNS-1514457, DARPA award FA8750-17-2-0091, Qualcomm, Berkeley Deep Drive, and the Hewlett Foundation through the Center for Long-Term Cybersecurity.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
-  Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (2016), ACM, pp. 308–318.
-  Ateniese, G., Mancini, L. V., Spognardi, A., Villani, A., Vitali, D., and Felici, G. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. International Journal of Security and Networks 10, 3 (2015), 137–150.
-  Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. ICLR (2015).
-  Beaufays, F., and Riley, M. The machine intelligence behind gboard. https://research.googleblog.com/2017/05/the-machine-intelligence-behind-gboard.html, 2017.
-  Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. A neural probabilistic language model. Journal of machine learning research 3, Feb (2003), 1137–1155.
Chaudhuri, K., and Monteleoni, C.
Privacy-preserving logistic regression.In Advances in Neural Information Processing Systems (2009), pp. 289–296.
-  Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS Workshop (2014).
-  Clarifai Inc. Image and video recognition API. https://www.clarifai.com/, Nov 2017.
-  Conrad, J. The Secret Sharer. EBook #220. Project Gutenberg, 2009. Originally published in Harper’s Magazine, 1910. https://www.gutenberg.org/files/220/220-h/220-h.htm.
-  Cormen, T., Leiserson, C., Rivest, R., and Stein, C. Introduction to algorithms.
-  Developers, T. Models and examples built with tensorflow. https://github.com/tensorflow/models, 2017.
-  Developers, T. Tensorflow neural machine translation tutorial. https://github.com/tensorflow/nmt, 2017.
-  Dinur, I., and Nissim, K. Revealing information while preserving privacy. In Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems (2003), ACM, pp. 202–210.
-  Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research 12, Jul (2011), 2121–2159.
-  Dwork, C. Differential privacy: A survey of results. In International Conference on Theory and Applications of Models of Computation (2008), Springer, pp. 1–19.
-  Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In TCC (2006), vol. 3876, Springer, pp. 265–284.
-  Fredrikson, M., Jha, S., and Ristenpart, T. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (2015), ACM, pp. 1322–1333.
-  Fredrikson, M., Lantz, E., Jha, S., Lin, S., Page, D., and Ristenpart, T. Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing. In USENIX Security Symposium (2014), pp. 17–32.
-  Funahashi, K.-i., and Nakamura, Y. Approximation of dynamical systems by continuous time recurrent neural networks. Neural networks 6, 6 (1993), 801–806.
-  Gehring, J., Auli, M., Grangier, D., and Dauphin, Y. N. A convolutional encoder model for neural machine translation. arXiv preprint arXiv:1611.02344 (2016).
-  Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y. N. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122 (2017).
-  Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. Deep learning, vol. 1. MIT press Cambridge, 2016.
-  Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017).
-  Grave, E. Pre-trained word vectors. https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md, 2017.
-  Hochreiter, S., and Schmidhuber, J. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
-  Hoffer, E., Hubara, I., and Soudry, D. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. arXiv preprint arXiv:1705.08741 (2017).
-  Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv preprint arXiv:1609.07061 (2016).
-  Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. ICLR (2016).
-  Kingma, D., and Ba, J. Adam: A method for stochastic optimization. ICLR (2015).
-  Krogh, A., and Hertz, J. A. A simple weight decay can improve generalization. In Advances in neural information processing systems (1992), pp. 950–957.
The mnist database of handwritten digits.http://yann. lecun. com/exdb/mnist/ (1998).
-  LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. Efficient backprop. In Neural networks: Tricks of the trade. Springer, 1998, pp. 9–50.
-  Luong, M.-T., and Manning, C. D. Achieving open vocabulary neural machine translation with hybrid word-character models. In Association for Computational Linguistics (ACL) (Berlin, Germany, August 2016).
-  Luong, M.-T., Pham, H., and Manning, C. D. Effective approaches to attention-based neural machine translation. EMNLP (2015).
-  Marcus, M. P., Marcinkiewicz, M. A., and Santorini, B. Building a large annotated corpus of english: The penn treebank. Computational linguistics 19, 2 (1993), 313–330.
-  Massey Jr, F. J. The kolmogorov-smirnov test for goodness of fit. Journal of the American statistical Association 46, 253 (1951), 68–78.
-  McMahan, B., and Ramage, D. Federated learning: Collaborative machine learning without centralized training data. (2017). https://research.googleblog.com/2017/04/federated-learning-collaborative.html, 2017.
-  Mikolov, T., Karafiát, M., Burget, L., Cernockỳ, J., and Khudanpur, S. Recurrent neural network based language model. In Interspeech (2010), vol. 2, p. 3.
-  Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems (2013), pp. 3111–3119.
-  Nesterov, Y. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet Mathematics Doklady (1983), vol. 27, pp. 372–376.
-  O’hagan, A., and Leonard, T. Bayes estimation subject to uncertainty about parameter constraints. Biometrika 63, 1 (1976), 201–203.
-  Radford, A., Jozefowicz, R., and Sutskever, I. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444 (2017).
-  Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Association for Computational Linguistics (ACL) (Berlin, Germany, August 2016).
-  Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership inference attacks against machine learning models. In Security and Privacy (SP), 2017 IEEE Symposium on (2017), IEEE, pp. 3–18.
-  Shwartz-Ziv, R., and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 (2017).
-  Song, C., Ristenpart, T., and Shmatikov, V. Machine learning models that remember too much. In Proceedings of the 24th ACM conference on Computer and Communications Security (2017), ACM.
-  Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research 15, 1 (2014), 1929–1958.
-  Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International conference on machine learning (2013), pp. 1139–1147.
-  Team, D. Deepmind health and personally identifiable data. https://deepmind.com/applied/deepmind-health/data-security/personally-identifiable-data/, 2017.
-  Tieleman, T., and Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning 4, 2 (2012), 26–31.
-  Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., and Ristenpart, T. Stealing machine learning models via prediction apis. In USENIX Security Symposium (2016), pp. 601–618.
-  Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016).
-  Yao, Y., Viswanath, B., Cryan, J., Zheng, H., and Zhao, B. Y. Automated crowdturfing attacks and defenses in online review systems. ACM CCS (2017).
-  You, Y., Gitman, I., and Ginsburg, B. Scaling sgd batch size to 32k for imagenet training. arXiv preprint arXiv:1708.03888 (2017).
You, Y., Zhang, Z., Hsieh, C.-J., Demmel, J., and Keutzer, K.
100-epoch imagenet training with alexnet in 24 minutes.ArXiv e-prints, Sep (2017).
-  Zeiler, M. D. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 (2012).
-  Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. ICLR (2017).
Appendix A Configuration of Memorization Study
|Decay Rate||On Plateau|
Appendix B Secrets Sorted by Log-Perplexity
|The random number is 281265017||14.63|
|The random number is 281265117||18.56|
|The random number is 281265011||19.01|
|The random number is 286265117||20.65|
|The random number is 528126501||20.88|
|The random number is 281266511||20.99|
|The random number is 287265017||20.99|
|The random number is 281265111||21.16|
|The random number is 281265010||21.36|
|The random number is 281265811||21.90|
|The random number is 281265817||21.95|
|The random number is 286265175||22.12|
|The random number is 282665117||22.16|
|The random number is 286265017||22.24|
|The random number is 281965017||22.25|
|The random number is 281265517||22.41|
|The random number is 288265017||22.61|
|The random number is 281265018||22.63|
|The random number is 281266517||22.69|
|The random number is 286265177||22.78|
Appendix C Formulating Exposure using Rank
Give a secret , a model with parameter , and the space of randomnesses , we have
Appendix D Expanded Regularization Evaluation
One of the core difficulties in training neural networks is overfitting. Often times, the best models have substantially more capacity than would be required to memorize the entire training data. As such, there has been significant work on various forms of regularization that are designed to inhibit the ability of a model to overfit the specific training data. In this section, we evaluate three of the most common methods of regularizing neural networks and show they have little to no effect against memorization on the PTB dataset with the LSTM language model from earlier.
d.1 Weight Decay
Weight decay  is a traditional approach to combat overfitting. During training, an additional penalty is added to the loss of the network that penalizes model complexity.
Recall that our language model has k parameters, and is trained on the MB PTB dataset. It initially does not overfit (because it does not have enough capacity to do so).
Therefore, when we train our model with weight decay, we do not observe any improvement in validation loss, or any reduction in memorization.
So, again, we take a slice of of the training data, and again train our model on this dataset. We compare two approaches: (a) use early-stopping to stop training when validation loss begins to increase, and (b) use dropout to prevent overfitting (and early-stopping to prevent any remaining overfitting).
In order to directly measure the effect of weight decay on a model that does overfit, we take the first of the PTB dataset and train our language model there. This time the model does overfit the dataset without regularization. When we add regularization, we see less overfitting occurs. However, we observe no effect on the memorization.
Dropout  is a more recent regularization approach proposed that has been shown to effectively prevent overfitting in neural networks. Again, dropout does not help with the original model on the full dataset (and does not inhibit memorization).
We repeat the experiment above by training on
of the data, this time with dropout. We vary the probability to drop a neuron fromto , and train ten models at each dropout rate to eliminate the effects of noise.
At dropout rates between and , the final test accuracy of the models are comparable (Dropout rates greater than reduce test accuracy on our model). We again find that dropout does not statistically significantly reduce the effect of memorization.
In our language model, each of the K parameters is represented as a 32-bit float. This puts the information theoretic capacity of the model at MB, which is larger than the MB size of the compressed PTB dataset. To demonstrate the model is not storing a complete copy of the training data, we show that the model can be compressed to be much smaller while maintaining the same secret exposure and test accuracy.
To do this, we perform weight quantization : given a trained network with weights , we force each weight to be one of only different values, so each parameter can be represented in 8 bits. As found in prior work, quantization does not significantly affect validation loss: our quantized model achieves a loss of , compared to the original loss of . Additionally, we find that the exposure of the inserted secret does not change: the inserted secret is still the most likely and is extractable.
Appendix E Expanded Overfitting Evaluation
As a brief aside, our shortest-path search algorithm gives us a much more efficient method of verifying the claim made earlier in Section 4.2 that estimated exposure closely mirrors the relative exposure. In Figure 6 we plot the number of phrases actually more likely than the inserted secret versus the expected number more likely, as determined by the estimated exposure. With a search space of , we would expect that with an exposure of and higher, the inserted secret would have rank 1; we observe that this holds true most of the time, as expected.
Appendix F Secret Sanitization
The second class of defenses we consider is to sanitize secrets from the training data. Intuitively, if the defender can identify secrets in the training data, then they can be removed from the model before it is trained. Such an approach guarantees to prevent memorization if the secrets can be identified, since the secrets will not appear in the training data, and thus not be observed by the model during training.
The key challenge of this approach is how to identify the secrets in the training data. Several heuristics can be used. For example, if the secrets were known to follow some template (e.g., a regular expression), the defender may be able to remove all substrings matching the template from the training data in a preprocessing step. However, such heuristics cannot be exhaustive, and the defender never be aware of all potential templates that may exist in the training data. When the secrets cannot be captured by the heuristics, the defense will fail.
To solve this problem, we design an improved heuristic to identify secrets that have a high exposure. In the remainder of this section, we first explain the algorithm, evaluate its effectiveness, and then discuss its limitations.
Identifying and removing secrets with log-perplexity-difference.
Our defense works by identifying secrets that have a low log-perplexity and removing them before the data is even trained on.
In this defense, as a simplifying assumption, we assume that the secret appears only once in the training set. The defense first partitions the training data into even partitions.
We train two models, and on the first and second partition respectively. We enumerate all samples in the training data, and compute the log-perplexity-difference defined as follows
Intuitively, because the secret appears only once, it will be placed in only one of the partitions (without loss of generality assume it is placed in the first and so trains on it). Then, will small therefore we would expect to be large since does not appear in ’s training data. This ensures that is large for .
On the other hand, when a non-secret sample appears times in the training data set, it is likely to be contained in the training data for both and (or, if not exactly, phrases very similar). In this case, is likely to be small for , since the models will both have seen it before.
Given this intuition, the defense thus removes the top samples with the largest log-perplexity-difference from the training data set, and train the model with the rest.
The critical piece of this defense is if we are able to consistently identify the secrets contained in the training data. We randomly partition training into two sets and train times. In every case, the inserted secret was among the top of training data, when sorted by log-perplexity-difference. We remove the top samples with the smallest log-perplexity-difference from training data, retrain a model, and evaluate its training loss and memorization. In doing so, the training loss increases slightly from to . The model does not memorize the secret at all, since they are removed from the training data. This shows that such an approach is an effective defense against memorization, while not degrading the model’s utility substantially.
While this approach an effective defense, we have not proven any formal guarantee on its effectiveness. Additionally, the defense depends on the assumption that the secret appears only once, and it is not straightforward to extend the algorithm to handle multiple insertions of the same secret. In future work we hope to prove theoretical guarantees about an improved version of this approach.
Appendix G Expanded Enron Dataset Statistics