1 Introduction
The advent of electronic health records has led to an explosion in longitudinal health data. This data can power comparative effectiveness studies, provide clinical decision support, enable retrospective research over realworld outcomes, and inform clinical trial design. However, longitudinal health data is often complex, unstructured, and highdimensional and thus untapped. Typically, limited labeled data is available for downstream tasks of interest, and labels can be prohibitively expensive to obtain: long records are tedious to synthesize, comprehension requires domain expertise, and patient privacy regulations limit datasharing across institutions (Bleackley and Kim, 2013; Xia and YetisgenYildiz, 2012). Fortunately, given the large amount of unlabeled data, selfsupervision is a promising avenue.
In selfsupervision, models are first pretrained to optimize an objective over unlabeled data, with the goal of learning representations that capture important semantic structure about the input data modality. For example, in masked language modeling for text, the model is trained to predict the identity of randomly masked tokens. Performing well at this objective should require a representation of sentence syntax and semantics. Once pretrained, selfsupervised representations can be used for downstream supervised tasks.
However, despite the success of selfsupervision across domains, the development of new selfsupervised objectives has been a largely heuristic endeavor.
Why does pretraining improve performance on downstream tasks? Whether this happens depends on both the selfsupervised objective and the downstream task itself. But the assumptions linking the selfsupervised objective to the downstream tasks of interest are rarely, if ever, made explicit.In this work, we design a selfsupervised objective with a particular class of data distributions and downstream tasks in mind. We aim to make explicit the type of distributions and downstream tasks on which we expect this method to work. We are interested primarily in the types of timeseries that arise in longitudinal health data, such as long sequences of clinical notes, insurance claims data, biomarker measurements, or combinations thereof. A key differentiating feature of these data is that a given trajectory can change quickly. For example, a patient may develop a new symptom between subsequent healthcare visits, or a certain biomarker value (e.g., blood pressure) may dramatically increase.
Additionally, these changes largely tend to be irreversible with respect to time. For example, once the word “metastasis” appears in a clinical note, nearly all subsequent notes tend to comment on the state of that metastasis (so the word “metastasis” appears in those notes as well). To train a good representation for certain downstream problems (e.g., “what is the patient’s current disease stage?”), the selfsupervised objective should attend to these changes, rather than suppress them.
These properties make such data distributions unsuitable for several existing selfsupervised objectives. For example, Franceschi et al. (2019) train a model so that the representation of each time segment is more similar to those of its subsegments than the representation of a randomly chosen segment from another trajectory. Similar techniques have been used to learn image representations from video: two subsequent video frames are likely to contain the same objects (Mobahi et al., 2009; Goroshin et al., 2015). These approaches are all similar to the idea of slow feature analysis (Wiskott and Sejnowski, 2002) for extracting representations of an input signal that change slowly over time. Representation learning techniques based on the ideas of slow feature analysis are appropriate for some downstream tasks and timeseries data types, such as the ones studied in the works above, but not, we argue, for data where the timeirreversible features are highly useful for downstream classification.
In this work, we introduce a selfsupervised objective called ordercontrastive pretraining (OCP). For each trajectory in the input data, we sample random pairs of time segments, switch the order for half of them, and train a model to predict whether a given pair is in the correct order (positives) or in the incorrect order (negatives).
This procedure is shown in Figure 1. OCP is very similar to an existing technique known as permutationcontrastive learning, or PCL (Hyvärinen and Morioka, 2017). PCL was also designed to take advantage of temporal dependence between features of the input signal to learn useful representations. The key difference between these two objectives is in the sampling of the negatives. Where the negatives in OCP are incorrectlyordered window pairs, the negatives in PCL are random window pairs from the same trajectory, and could be in the correct order. In their simplest forms, the positive samples for the two methods are identical: pairs of consecutive windows in the correct order.
Intuitively, the same timeirreversible features that are useful for the OCP and PCL objectives should also be useful for downstream prediction tasks. To formalize and quantify this, we study a class of data distributions motivated by the preceding discussion. When the representation belongs to a simple hypothesis class (effectively, when the representation is a feature selector), we prove a finitesample bound on the downstream error of a representation learned using OCP. Although this setting is much simpler than those that appear in similar work (it involves linear, rather than nonlinear, representations of the input data), we show that this model still admits interesting behavior. In particular, we give an example of a data distribution in this setup where OCP and PCL provably learn different representations. Additionally, this model indicates that even when two methods have the same performance with infinite unlabeled data, there is an unlabeledsamplecomplexity benefit to using a “clean” distribution of negatives, which matches well with prior work on other contrastive learning algorithms (Chuang et al., 2020).
We supplement this motivating theoretical study with experiments on realworld timeseries data. Our results indicate that for the types of data and tasks discussed above, both OCP and PCL representations can enjoy better downstream prediction performance than those trained using existing selfsupervised baselines. Moreover, complementing our theoretical results, we show a realworld scenario where OCP outperforms PCL in the low labeleddata regime despite the seemingly minor difference between the two objectives. Given that OCP and PCL only differ slightly in their negative sampling, these results give further theoretical and empirical evidence for the importance of the negative sampling details in contrastive learning, complementing several recent works (Chuang et al., 2020; Robinson et al., 2021; Liu et al., 2021).
2 Orderpretraining algorithm
We suppose each data point is a time series, , where is the number of sample points and may vary with . We also suppose the samples take values in some common set . Let a window be an element of , and let be the corresponding element of .^{1}^{1}1For simplicity, we only consider windows of size 1. Our results straightforwardly generalize to windows of arbitrary size , where is a subinterval of and .
Given a trajectory , we use the following generative process to sample a data point for our contrastive task. First, is chosen uniformly at random from . Next, random windows and are chosen (in a manner explained below). The segments and corresponding to windows are combined into a tuple . The pair is then a sample for the contrastive task. A model
, given by a composition of a classifier
and a representation , is trained to predict from :(1) 
Here . That is, first computes the representation for each window, then uses a classifier to predict whether the tuple is in the correct order. The representation can then be reused on a downstream task. The remaining design choice is to specify the process for sampling windows.
Ordercontrastive pretraining.
A simple choice for sampling random windows , is to sample a random pair in the correct order when , and in the incorrect order when . We refer to the optimization problem (1) with this choice of sampling as ordercontrastive pretraining (OCP). This can easily be generalized to nonconsecutive window pairs. The pretraining task (1) is thus to contrast windows in the correct order with windows in the incorrect order.
Permutationcontrastive learning.
Another simple choice is to again sample a pair in the correct order when , but sample a random pair when . This is the data generation process for permutationcontrastive learning (Hyvärinen and Morioka, 2017). Note that the only differences between OCP and PCL are that in PCL, (i) the negative samples ( need not be consecutive, and (ii) some are in the correct order. The distributions of positive samples are identical. We refer to the procedure (1) with this sampling as permutationcontrastive learning (PCL). This exactly matches the contrastive sample distribution in Hyvärinen and Morioka (2017, equations (10)(11)). Here, the pretraining task is to contrast consecutive windows in the correct order versus random window pairs.
Comparison.
These two sampling methods seem very similar—they only differ slightly in the distribution of negatives (i.e., conditioned on ). However, we show theoretically and empirically in the following sections that they can learn very different representations when used in (1), and they can have different unlabeled sample complexities even if they eventually find the same representation. This gives further evidence of the importance of negative sampling for contrastive learning methods (see, e.g., Chuang et al. (2020)). We give a finitesample bound for the downstream classification performance of a representation learned using OCP in a simple setup motivated by time series data and predictive tasks in healthcare.
3 Finitesample guarantee for timeirreversible features
In this section, we study a class of distributions motivated by applications to timeseries data in healthcare. We assume for simplicity that each . We identify a simple set of three assumptions for which we can prove a finite sample guarantee for the set of feature selector hypotheses.
Assumptions (informal):

There exists a set of timeirreversible features. Formally, , .

When the features in are not changing, the other features are timereversible. More formally, for all , and all , if , .

There are no “redundant” features in .
The precise statements of these assumptions are located in Appendix A (Assumptions A.1A.3). Intuitively, the first two assumptions guarantee that the featureset is an optimal choice of representation for the OCP pretraining objective (when is the class of feature selector representations), and the third (more technical) assumption guarantees that the optimum is unique.
When these assumptions are satisfied and the features are the correct features for downstream classification, we prove a finitesample bound for a model pretrained using OCP. The bound only depends on the VCdimension of the downstream hypothesis class, , rather than on .
Like some results in the nonlinear ICA literature (e.g., Hyvärinen and Morioka (2017)) our results only apply in the regime where there is enough unlabeled data to identify the “correct” representation. It’s then immediate that only factors in to the labeleddata dependence. However, our model also allows us to give upper bounds on the amount of unlabeled data required to reach that regime. This allows us to more rigorously study other aspects of contrastive pretraining, such the role of bias in the negative distribution, which has been shown to affect the performance of other contrastive learning algorithms (Chuang et al., 2020).
We now give a simple example of a class of distributions satisfying these assumptions, grounded in our running application of health timeseries data. We prove in Theorem A.1 that this example satisfies assumptions A.1A.3. Despite its simplicity, our findings suggest that this model allows for several interesting phenomena that also occur in practice, which could make it useful for further study of contrastive learning methods on timeseries.
3.1 Extraction example
A common task in clinical informatics is to extract for each time the patient’s structured disease stage, which enables downstream clinical research (Kehl et al., 2019, 2020). Each time point could be an encoding of the clinical note from a patient’s visit at time . Let be the observed label for time point . The end goal is to train a model over a representation to minimize the downstream risk:
Here we make a prediction for every time point, and the expectation is over the time index as well as the trajectory .
Model.
For each , we denote by
the random variable corresponding to indices
at time . Suppose the set of feature indices is partitioned into three types of features:
A set of timeirreversible features. We also assume that each has a nonzero probability of activating on its own, without the other features in . That is, for each there exists with . This ensures that assumption A.3 is satisfied. Such features include the onset/progression of chronic conditions and markers of aging (Pierson et al., 2019). For example, appearance of the word “metastasis” in a clinical note.

Noisy versions of : for each , there exists with , with , for all . Additionally, is conditionally independent of the other variables (for all times) given its parent variable . For example, the presence of certain interactions with the health system—such as deciding to attend physical therapy—may be a noisy reflection of the patient’s true disease state, which is captured by .

Background, reversible features : features such that for all and all ,
Consider, for example, common words such as “and”, “chart”, etc., in a clinical note, whose presence or absence gives no order information.
Note that we do not make any independence assumptions between the features in this example other than the ones mentioned above.
3.2 Finite sample bound
Suppose we observe a large set of unlabeled data points drawn independently from the marginal distribution of , and a much smaller set of labeled data
drawn independently from the joint distribution of
and the downstream label (now we use to denote the downstream label rather than the pretraining label).Let the representation hypothesis class . That is, our representation will select features to be used downstream. We identify each set of indices with the mapping given by projection onto those indices. Let be the downstream hypothesis class, with .^{2}^{2}2Assume is closed under permutations of the input dimensions. This ensures that we only need to identify the features belonging to , and don’t need to put them in a particular order to do well at downstream prediction. The set of linear hypotheses has this property. The mapping first selects the input features represented by , then passes the values of these features through .
Our end goal is to design a learning algorithm with a downstream excess risk bound. If is the hypothesis output by , we want an upper bound on the excess risk: If we let be empirical risk minimization (ERM) over on the small labeled sample (i.e., the method directly optimizing the downstream objective over without pretraining),^{3}^{3}3Assume for simplicity that for each trajectory , a single time is chosen uniformly at random and are passed to the learner, so the learner sees i.i.d. samples. A more detailed treatment would handle the dependence between multiple time points to get bounds that decrease as when possible (e.g., Mohri and Rostamizadeh, 2010). a standard result (e.g., ShalevShwartz and BenDavid (2014, Theorem 6.8)) implies:
(2) 
with high probability over the sampling of the data. On the other hand, let be the algorithm that first uses unlabeled data to pretrain a representation by minimizing (1), then minimizes the downstream risk over (i.e., a 2phase ERM learner). The following theorem states that under Assumptions A.1A.3, we can give a more parsimonious upper bound on the excess risk.
Theorem 3.1 (informal).
Assume that Assumptions A.1A.3 hold and that , i.e., that the optimal downstream classifier is a function of the features . Let be the hypothesis output by the pretrain/finetune learner using OCP for pretraining. Then for large enough, satisfies:
(3) 
with high probability over the sampling of the data.
The pretrain + finetune bound (3) has a better dependence on the labeled dataset size than the downstream ERM learner bound (2). The large unlabeled dataset allows for the learning of a good representation
without using any labeled data. Even in this simple feature selection setting, this bound may be much tighter than the directdownstream bound when
is a fairly complex hypothesis class and . Even for linear, is roughly (see e.g. (Abramovich and Grinshtein, 2018)), so (3) can even save over (2) in this case. In fact, we show in Section 4 that OCP can improve the performance of sparse linear models in a realworld lowlabeled data setting (compared to direct downstream prediction without pretraining).In this section we gave a simple example of a class of distributions, together with an assumption linking the distribution to the downstream task (the timeirreversible features are the most useful ones for downstream classification) for which we can prove that OCP gives a more parsimonious bound on the labeled sample complexity. While the example in Section 3.1 seems straightforward, we show now that it still admits interesting behavior. In particular, there are data distributions that satisfy Assumptions A.1A.3, but where PCL and OCP learn different representations.
PCL versus OCP: different infinitedata optima.
There are examples of the model from Section 3.1 where PCL and OCP learn provably different representations even with infinite unlabeled samples, despite the minor difference in their sampling schemes. Intuitively, the existence of a periodic feature (such as a procedure always performed at a particular time of day) is strongly predictive of whether two samples are consecutive, but need not be predictive of whether a pair of consecutive samples are in the correct order. Concretely, consider a feature such that , and . Inclusion of this feature doesn’t violate assumptions A.1A.3—indeed, would qualify as a “background” feature under our model—so Theorem 3.1 guarantees that OCP finds the correct representation. However, in PCL, every nonconsecutive sample is a negative. But only nonconsecutive samples can have , so is helpful for the PCL objective. We treat this example more formally in Appendix A, but our synthetic results in Section 4 also show that a background periodic feature can affect the PCL representation.
“Debiased” negatives.
PCL has some negatives that are actually in the correct order. Prior work on contrastive learning has called this “bias” in the negative distribution (Chuang et al., 2020). What’s the role of this “bias?” Does it affect the learned representations? Does it affect the amount of unlabeled data required to find a good representation? For distributions satisfying assumptions A.1A.3 and when is the class of featureselectors, we answer these questions in the negative and positive, respectively.
In particular, consider the analogue of OCP that instead of always choosing when ( as used in OCP, not the downstream label), instead just chooses a random pair with
(i.e., a random consecutive pair). We refer to this as OCPbiased, since some of the negatives are actually in the correct order. However, the following theorem shows the estimator obtained by minimizing this objective is
not biased in a statistical sense:Theorem 3.2 (informal).
However, it does affect the bound on unlabeled sample complexity required to obtain a good representation:
Proposition 1 (informal).
The upper bound on the sample complexity required for OCPbiased to identify is worse than the upper bound for OCP.
4 Experiments
4.1 Synthetic data
We demonstrate the importance of negative sampling over two synthetic datasets from the model in Section 3. Each distribution contains alongside a number of noisy features. We generate pretraining datasets of different sizes (50 to 16,000) and sample pairs from each dataset according to OCP, PCL, and OCPbiased. We then conduct an L0 regression over the sample pairs and analyze how many variables in were correctly recovered. The top panel of Figure 2 shows a distribution where PCL does not recover in the infinite data limit—this distribution includes a periodic background feature in that is selected by PCL. The bottom panel of Figure 2 shows a distribution where PCL is able to recover all of , but requires a larger sample complexity than OCP. In both cases, OCP and OCPbiased find the same representation, but the former has better dependence on unlabeled data. We provide the details and explanations for these experiments in Appendix B.
4.2 Realworld data
We show OCP yields significant improvements in the lowlabel regime on extraction from clinical notes.
Progression dataset.
We utilize a dataset of fully deidentified clinical notes from a single institution. This research was reviewed by our institution’s Committee on the Use of Humans as Experimental Subjects and determined to be IRBexempt. The dataset contains data for 82,839 patients with cancer, with a median of 12 radiology notes each. Each radiology note focuses on one body area (e.g., chest CT scan). In addition, we have a subset of 135 patients with progressive lung cancer with 1095 labeled radiology notes. Each note was labeled posthoc by an oncologist as ‘indicating progression’ (19%), ‘not indicating progression’ (79.5%), or ‘ambiguous’ (1.5%).
Mean notelevel AUC of regularized logistic regression over different dataset sizes. Averaged over the 5 folds, performance was optimal for each dataset size when restricted to the features with nonzero coefficients recovered by OCP. 
Experimental setup.
We investigate extraction of these binary progression labels from the Impression section of the note. The labeled data was split via 5fold crossvalidation: each fold contained sets of sizes 64% (train), 16% (validation), and 20% (test); for a given fold, no patient examples were ever split between sets. On each fold, we used the test set to benchmark models trained using different amounts of the labeled training data: from just 5 training patients () to all of the training patients. We excluded patients with downstream labels from pretraining. For contrastive pretraining schemes, a pretraining window pair was sampled once per each unique body area (e.g. chest, brain) that was scanned at least twice, capped at five locations per patient. This resulted in 158,000 samples for pretraining.
Pretraining for feature selection.
We first leverage a linear model to evaluate our modeling assumptions from Section 3. We compare downstream progression extraction performance of (i) a vanilla logistic regression model and (ii) a logistic regression model only using the features selected by OCP. We test on all five folds for five training dataset fractions.
For each experiment, our dataset is featurized using the unigrams and bigrams that occur in at least 5% of the labeled training data set. They are vectorized using the
term frequencyinverse document frequency weighting scheme, via scikitlearn (Pedregosa et al., 2011, BSD 3clause license). We conduct feature selection as an optional intermediate step preceding progression extraction. For OCP, we train a L1 regression model over the 158,000 pretraining pairs of consecutive radiology notes. The regularization constant was set such that there were features with nonzero weights. In addition to OCPderived features, we select the 50 most common features, and 5 random subsets of 50 features to serve as a comparison.We train scikitlearn logistic regression models for downstream progression extraction over each feature set; further details are in Appendix C. Results can be seen in Figure 3a. Even with a simple bagofwords representation, feature selection with OCP outperforms directly training a tuned logistic regression model on the available labeled data (“All features”), especially for small dataset sizes. A paired test finds that the model with OCPselected features is significantly better than the directdownstream model on a sixteenth of the data (). Note that selecting the most common features or a random set of features does not compare, showing that OCP does not improve performance by simply reducing the feature dimension in a redundant space.
We manually examined the OCPselected features and their coefficients (Figure 2(b)). The features included (e.g. increased, decreased) strongly indicate disease progression, while those discarded (e.g. discussed) largely seem to be noise. Of the nonzero coefficients, 76% have a positive weight; this indicates that the pretraining model focuses mostly on features that have been turned on to conduct the ordering task, fitting with our motivating theoretical setting.
Fraction of training data  

AUC diff. (OCP Win %)  1  1/2  1/4  1/8  1/16 
OCP AUC  0.87 .03  0.86 .04  0.84 .04  0.82 .03  0.81 .03 
OCP BERT  0.08 (93%)  0.12 (100%)  0.12 (100%)  0.18 (100%)  0.22 (100%) 
OCP FT LM  0.03 (80%)  0.04 (82%)  0.04 (82%)  0.08 (93%)  0.10 (89%) 
OCP PtContrastive  0.03 (86%)  0.03 (77%)  0.05 (91%)  0.09 (91%)  0.12 (97%) 
OCP PCL  0.00 (53%)  0.00 (46%)  0.03 (64%)  0.03 (76%)  0.06 (87%) 
Nonlinear representations.
We now study the use of OCP for pretraining nonlinear representations. We compare performance of a BERT model pretrained using OCP to several other selfsupervision methods. We investigate the BERT base model and the BERT base model after it is pretrained using: (i) FT LM: finetuned masked language modeling over an equivalent number of impressions, (ii) PtContrastive: a patientlevel contrastive objective (identical positive sampling to OCP and PCL, but each negative is a random note of the same note type from a different patient, similar to Diamant et al. (2021)), (iii) PCL: contrastive pretraining with PCL sampling (each negative is a random pair of notes of the same type from the same patient), (iv) OCP: contrastive pretraining with OCP sampling (each negative is a pair of notes of the same type in the incorrect order). All pretraining is conducted over three seeds, and all three contrastive objectives were trained with the same number of pairs (158,000). Implementation for language modeling and contrastive pretraining came from Wolf et al. (2020, Apache2.0 License) with full details in Appendix C. After model pretraining/finetuning, the selfsupervised representation layers were frozen, and a single L2regularized linear layer was added on top. The goal of freezing was to isolate the effect of pretraining to understand representation quality, due to the instability of training BERT on small downstream tasks (Zhang et al., 2020).
Results can be seen in Table 1. The top row shows that OCP has only a modest drop in performance even when trained on the data from just 5 patients (). Since correlations exist in AUC across the 5 folds, 3 seeds, and 5 dataset sizes, standard statistical comparison testing is inappropriate. Instead, we present the mean increase in AUC from OCP, as well as the percentage of the time OCP outperformed the comparisons. Unsurprisingly, BERT alone (trained on nonclinical text) unsurprisingly does not perform well outofthebox; finetuning with language modeling improves performance, but still suffers in the low data regime. Among the contrastive objectives, the crosspatient objective is the weakest, which may follow since its pretraining task was the easiest (82% accuracy on validation). It could rely on features that differed between patients, instead of being forced to focus on the temporal features that differed within a patient’s timeline. PCL performs equivalently to OCP at large data sizes, but at the smaller data set sizes, it loses to OCP a large majority of the time.
5 Related work
Order pretraining.
Others have found orderbased selfsupervision useful for more complex timeseries data, but without theoretical study. For example, learning the order of frames within a video yields representations useful for downstream activity classification (Fernando et al., 2015; Misra et al., 2016; Lee et al., 2017; Wei et al., 2018). Most similar to our work, Hyvärinen and Morioka (2017) introduced permutationcontrastive learning (PCL) and proved nonlinear identifiability for representations learned using PCL in an ICA setting. That is, they gave distributional conditions where PCL provably recovers the “correct” nonlinear representation of the input given infinite unlabeled data. Our theoretical and empirical results indicate that there can be nontrivial differences between OCP and PCL’s downstream performance. Deeper understanding of what data distributions and downstream tasks are “right” for PCL versus OCP (and for other contrastive sampling methods) is an interesting direction for future theoretical study.
Pretraining for medical timeseries.
Several other pretraining objectives have been explored on clinical timeseries data. A contrastive learning setup similar to our patientcontrastive baseline has shown promising results on electrocardiograph signals (Diamant et al., 2021; Kiyasseh et al., 2020). Banville et al. (2021) studied a contrastive objective for electroencephalography signals, in which windows of a signal are judged to be similar if they occur within a certain time gap, and dissimilar if they are far away in time. Intuitively, this objective is wellsuited to data where the true representation “changes slowly” with time, as with Franceschi et al. (2019) (discussed in Section 1). Other objectives include autoencoding (Fox et al., 2019) and masked prediction over text and tabular data, to mixed results (Steinberg et al., 2020; Huang et al., 2019; Yoon et al., 2020; McDermott et al., 2021). Multitask pretraining supplies improvements, but unlike our work, it relies on additional labeled data from closelyrelated downstream tasks (McDermott et al., 2021).
Selfsupervision theory.
Like our work, Arora et al. (2019); Liu et al. (2021) and Tosh et al. (2020, 2021) give downstream finitesample error bounds for representations learned using particular contrastive learning objectives. Our motivating theoretical setting and proof techniques are simpler than the ones considered in these works, but we show that our setup in Section 3 is (i) complex enough to allow for some of the same nontrivial behavior observed by contrastive methods in practice (Sections 3.2, 4.1) and (ii) it has some practical applications (Section 4.2).
6 Limitations and Conclusion
We have shown both theoretically and empirically that ordercontrastive pretraining is an effective selfsupervised method for certain types of timeseries data and downstream tasks. On realworld longitudinal health data, we find that representations from OCP significantly outperform others, including the similar PCL, in the small data regime. However, OCP is not always suitable. For example, cases of temporal leakage (e.g., the date in a note) can lead to weak OCP (and PCL) representations downstream, since they provide a shortcut during pretraining. While dates are straightforward to censor, more complex global nonstationarities irrelevant to downstream tasks would present a challenge to these methods.
Our theoretical setup and results in Section 3 also serve to highlight that contrastive pretraining methods can be very sensitive to the precise sampling details, and provide a simple model for studying these details that is still complex enough to capture some empirical phenomena. This suggests that obtaining broader theoretical guidelines for selecting a contrastive distribution is an interesting direction for future work.
References
 Highdimensional classification by sparse logistic regression. IEEE Transactions on Information Theory 65 (5), pp. 3068–3079. Cited by: §3.2.
 A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229. Cited by: §5.
 Uncovering the structure of clinical eeg signals with selfsupervised learning. Journal of Neural Engineering 18 (4), pp. 046020. Cited by: §5.
 The merit and agony of retrospective chart reviews: a medical student’s perspective. British Columbia Medical Journal 55, pp. 374–375. Cited by: §1.
 Debiased contrastive learning. In NeurIPS, Cited by: Appendix A, §1, §1, §2, §3.2, §3.
 Patient contrastive learning: a performant, expressive, and practical approach to ecg modeling. arXiv preprint arXiv:2104.04569. Cited by: §4.2, §5.

Modeling video evolution for action recognition.
In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, pp. 5378–5387. Cited by: §5.  Learning through limited selfsupervision: improving timeseries classification without additional data via auxiliary tasks. Cited by: §5.
 Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems, Vol. 32, pp. . External Links: Link Cited by: §1, §5.
 Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pp. 4086–4093. Cited by: §1.
 Clinicalbert: modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342. Cited by: §5.
 Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pp. 460–469. Cited by: §1, §2, §3, §5.

Assessment of deep natural language processing in ascertaining oncologic outcomes from radiology reports
. JAMA oncology 5 (10), pp. 1421–1429. Cited by: §3.1.  Natural language processing to ascertain cancer outcomes from medical oncologist notes. JCO Clinical Cancer Informatics 4, pp. 680–690. Cited by: §3.1.
 Clocs: contrastive learning of cardiac signals. arXiv preprint arXiv:2005.13249. Cited by: §5.
 Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision, pp. 667–676. Cited by: §5.
 Contrastive learning of strongmixing continuoustime stochastic processes. In International Conference on Artificial Intelligence and Statistics, pp. 3151–3159. Cited by: §1, §5.
 A comprehensive ehr timeseries pretraining benchmark. In Proceedings of the Conference on Health, Inference, and Learning, CHIL ’21, New York, NY, USA, pp. 257–278. External Links: ISBN 9781450383592, Link, Document Cited by: §5.
 Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision, pp. 527–544. Cited by: §5.
 Deep learning from temporal coherence in video. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 737–744. Cited by: §1.
 Foundations of machine learning. MIT press. Cited by: §A.2.
 Stability bounds for stationary mixing and mixing processes.. Journal of Machine Learning Research 11 (2). Cited by: footnote 3.
 Scikitlearn: machine learning in Python. Journal of Machine Learning Research 12, pp. 2825–2830. Cited by: §C.1, §4.2.
 Inferring multidimensional rates of aging from crosssectional data. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 97–107. Cited by: 1st item.
 Contrastive learning with hard negative samples. ICLR. Cited by: §1.
 Understanding machine learning: from theory to algorithms. Cambridge university press. Cited by: §A.2, §3.2.
 Language models are an effective patient representation learning technique for electronic health record data. arXiv preprint arXiv:2001.05295. Cited by: §5.
 Contrastive estimation reveals topic posterior information to linear models. arXiv preprint arXiv:2003.02234. Cited by: §5.
 Contrastive learning, multiview redundancy, and linear models. In Algorithmic Learning Theory, pp. 1179–1206. Cited by: §5.
 Learning and using the arrow of time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8052–8060. Cited by: §5.
 Slow feature analysis: unsupervised learning of invariances. Neural computation 14 (4), pp. 715–770. Cited by: §1.
 Transformers: stateoftheart natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, pp. 38–45. External Links: Link Cited by: §C.2, §4.2.
 Clinical corpus annotation: challenges and strategies. In Proceedings of the Third Workshop on Building and Evaluating Resources for Biomedical Text Mining (BioTxtM’2012) in conjunction with the International Conference on Language Resources and Evaluation (LREC), Istanbul, Turkey, Cited by: §1.

VIME: extending the success of selfand semisupervised learning to tabular domain
. Advances in Neural Information Processing Systems 33. Cited by: §5.  Revisiting fewsample bert finetuning. arXiv preprint arXiv:2006.05987. Cited by: §4.2.
Appendix A Theory details
In this section, we provide proofs for the simple model introduced in Section 3. In particular, we prove the finite sample bound for OCP by first proving that the driver features are the unique optimal representation, and then we apply standard uniform convergence arguments to obtain the bound (3).
We also show how socalled bias (Chuang et al., 2020) in the distribution of negative samples can affect OCP. In particular, suppose that instead of choosing windows as positive and as negatives, we chose as positives and a random pair with as negatives. This is analogous to PCL’s negative sampling, where some of the negatives are still in the correct order. The difference with PCL is just that all negatives are still consecutive. For the model in Section 3, we prove that (i) in the infinitedata limit, this “biased” version recovers the same representation as (so in fact, the estimator obtained by minimizing this objective is not biased in a statistical sense) and (ii) the bound on unlabeled sample complexity required to find a good representation is worse for this biased version of OCP than for the unbiased version. This gives more theoretical evidence for the value of debiasing the negative distribution in contrastive learning, where possible: even if it doesn’t change the representation learned with infinite unlabeled data, debiasing the contrastive distribution can improve unlabeled sample complexity.
a.1 Assumptions and example class of distributions
Here we provide simple assumptions under which a set of features is the unique optimal solution to (1) for the model from Section 3. For simplicity in this section, we only consider consecutive windows and the case where (the sequence length) is the same for all sequences, but all of our results generalize (with suitable modifications to these assumptions) to the case where the positive and negative distributions over windows are symmetric up to ordering of the elements (correct versus incorrect), and to the case with a distribution over .^{4}^{4}4For example, if for all and , (i.e., the distribution of any window pair only depends on the fact that the trajectory is still active, and not on the actual length) then assumptions remain the same.
Assumption A.1.
For all and all ,
Assumption A.2.
For all and with , and for all ,
Assumption A.3.
For all such that , there exists and , , with and:
Now we prove that the example from Section 3 satisfies these assumptions.
Theorem A.1.
Partition the indices into three sets satisfying the following assumptions:

[beginpenalty=10000]

Timeirreversible features satisfying Assumption A.1. We also assume that each has a nonzero probability of activating on its own, without the other features in . That is, for each there exists , with .

Noisy versions of : for each , there exists with , with , for all . Additionally, is conditionally independent of the other variables (for all times) given its driver : , where if and otherwise.

Background, reversible features : features such that for all and all ,
Proof.
Assumption A.1 is satisfied by definition. For Assumption A.2, fix with and . Then we have:
Using the conditional independence assumption for the features in , the righthandside factors to:
Because , trivially this is equal to:
The reversibility of the features in (and summing over the full joint distribution) imply that the above is equal to:
Finally, note that because , and are identically distributed by the definition of features in , so:
Combining with the previous equation and simplifying, we obtain
which is Assumption A.2.
Finally, for Assumption A.3, fix with , and fix some . We assumed that each has some probability of activating on its own, so there exists with . In particular, this implies that there exist with , , and . By summing over variables in this joint distribution, that implies , so Now we need to reverse and for the indices to account for the other term in the of Assumption A.3. For each , we can take while maintaining and , since we assumed that all values of . We know that , so we will try to rearrange to make use of this fact by attempting to switch for .
We have:
because . Let be the set of indices in corresponding to noisy indicators of feature . Because we chose so that , all features in are identically distributed at times and . Therefore, we can switch and for some of the indices in :
By the definition of the features in , we can also switch and for . The only remaining terms to switch are and . Because we took for all without loss of generality, and , , we have:
Hence, we can finally combine:
So we have shown for these that:
But we know the RHS is positive because had . Therefore,
So we have shown that both and are strictly positive, which gives Assumption A.3. ∎
a.2 Optimal representations and finitesample guarantee
This first lemma shows that the feature set is an optimal representation for the OCP objective.
Lemma A.1.
Let
be the Bayesoptimal classifier for the OCP objective. There exists a classifier , depending only on the coordinates of and , achieving the same OCP error as .
Proof.
First we compute the error of the Bayesoptimal classifier. For , , we say if . Assumption A.1 implies that:
That is, given windows , if the set of variables active in is properly contained in the set of variables active in , we know the windows must be in the correct order. Likewise, if the active variables in are properly contained in those of , the windows must be in the wrong order. Assumption A.1 also implies that the only possible cases are , , and . For the last case, we have:
Assumption A.2 implies that the two terms in the denominator are equal when
Comments
There are no comments yet.