SMD
None
view repo
Extracting an understanding of the underlying system from high dimensional data is a growing problem in science. One primary challenge involves clustering the data points into classes with interpretable differences. A challenge in clustering high dimensional data is that multimodal signatures that define clusters may only be present in a small but unknown subspace. Discovering the subspace that defines clusters provides low dimensional representations of the data which capture cluster diversity, and provides greater understanding of the system by identifying the key underlying variables. Here, we define a class of problems in which linear separability of clusters is hidden in a low dimensional space. We propose an unsupervised model-free method to identify the subset of features that define a low dimensional subspace in which clustering can be conducted. This is achieved by averaging over discriminators trained on an ensemble of proposed cluster configurations. We then apply our method to single cell RNA-seq data from mouse gastrulation, and identify 27 key transcription factors (out of 409 total), 18 of which are known to define cell states through their expression levels. In this inferred subspace, we find clear signatures of known cell types that eluded classification prior to discovery of the correct low dimensional subspace. This method can be used as a wrapper for existing clustering algorithms to find low dimensional informative subspaces with multimodal signatures within high dimensional data.
READ FULL TEXT VIEW PDFNone
Multimodal-Mapping of high dimension sparse matrix project
Recent technological advances have resulted in a wealth of high dimensional data in biology, medicine, and the social sciences. In unsupervised contexts where the data is unlabeled, inferring distinct classes and identifying their characteristics is a key step towards mechanistic insight. This process of clustering relies on identifying modes of the data distribution which correspond to different classes. Clustering in high dimensions is difficult, however, both because of unavoidably low data density in high dimensions (the “curse of dimensionality"
[1]) and because of the possibility that a small but unknown fraction of the measured features define the true clusters while the remaining features are uninformative of cluster identity [2, 3].Identifying informative features has long been of interest in the statistical literature. When the data is labeled, allowing for a supervised analysis, there are successful techniques for extracting important features using high dimensional regressions. When there is no labeled training data, unsupervised discovery of features relevant for clustering is difficult. Existing methods to do so rely either on finding features that correlate with principal components [4], which can be misleading when these components do not capture the variation relevant to cluster identity [3], or on optimizing cost functions which require knowledge of the number of clusters [2]. When little is known about the underlying system, and no forward model exists, an unbiased discovery of key features to define clusters and hence classes is essential for extracting insight.
Using gene expression data to understand processes in developmental biology highlights this challenge. In a developing embryo, multi-potent cells make a sequence of decisions between different cell fates, eventually giving rise to all the differentiated cell types of the organism. The goal is both to determine the physiological and molecular features that define the diversity of cell states, and to uncover the molecular mechanisms that govern the generation of these states. Decades of challenging experimental work in developmental biology suggests that a small fractions of genes control specific cell fate decisions [5, 6, 7]. Recent experimental techniques measure tens of thousands of features – gene expression levels – from individual cells obtained from an embryo over the course of development, producing high dimensional data sets [8, 9]. Clustering these data to extract cell states and identifying the small fractions of key genes that govern the generation of cellular diversity during development has been difficult [10, 11]. However, mapping cellular diversity back to specific molecular elements is a crucial step towards understanding how gene expression dynamics lead to the development of an embryo.
Here we show that as the fraction of relevant features decreases, existing clustering and dimensionality reduction techniques fail to discover the identity of relevant features. We show that when the linear separability of clusters is restricted to a subspace, the identity of the subspace can be found without knowing the correct clusters by averaging over discriminators trained on an ensemble of proposed clustering configurations. We then apply it to previously published single-cell RNA-seq data from the early developing mouse embryo [12], and discover a subspace of genes in which a greater diversity of cell types can be inferred. Further, the relevant subspace of genes that we discover not only cluster the data but are known from the experimental literature to be instrumental in the the generation of the different cell types that arise at this stage. This approach provides unsupervised sparse feature detection to further mechanistic understanding and can be broadly applied in unsupervised data analysis.
To understand how the decreasing fraction of relevant features affects data analysis, consider data from classes in a space with features. Assume that can be partitioned into two subspaces. First, an informative subspace of dimension , in which the clusters are separable. And second, an uninformative subspace with dimension in which the clusters are not separable. An example of such a distribution is shown in Figure 1 with two clusters, and .
The correlation between the distances computed in the full space with that in the relevant subspace scales as (see, 4.1). When the fraction of relevant features is small, or equivalently , correlations between samples become dominated by noise. In this regime, without the correct identification of
, unsupervised analysis of the data is difficult, and typical dimensionality reduction techniques (PCA, ICA, UMAP, etc) fail. We demonstrate this by constructing a Gaussian mixture model with 7 true clusters which are linearly separable in a subspace
with dimension , and drawn from the same distribution (thus not linearly separable) in the remaining dimensions. As the ratio increases, the separability of the clusters in various projections decreases (Figure 2). For the same reason, clustering methods and graph embeddings that rely on Euclidean distances between data points to measure similarity deteriorate as increases (Figure S1).In many cases, identifying the “true" may be challenging. However, eliminating a fraction of the uninformative features and moving to a regime of smaller
could allow for more accurate analysis using classical methods. We next outline a method to weight dimensions to construct an estimate of
and to reduce .To develop a framework to identify the relevant features, consider data where samples are represented in the measurement basis . Assume that the data is structured such that each data point is a member of one of clusters, . Let be the subspace of in which the data points belonging to the pair of clusters , are linearly separable. Let
be unit vector normal to the max-margin hyperplane separating clusters
and . In the space orthogonally complement to , the two clusters are not linearly separable. One can similarly define such subspaces and associated hyperplanes , one for each pair of clusters in . We define a weight for each dimension by it’s component on the s:(1) |
Knowing the cluster configuration would allow us to directly compute
by finding max-margin classifiers and using Equation
1. Conversely, knowing would allow for better inference of the cluster configuration because restriction to a subspace in which would move to a regime of smaller (Figure 2). Existing work has focused on finding and simultaneously, through either generative models or optimizing a joint cost function. Such methods either rely on context specific forward models, or tend to have problems with convergence on real data sets (see 2.7 and ref. [2]).We focus here on estimating when is unknown. We consider the limit in which the dimensions of each , take on the smallest possible value of , which maximizes the ratio for all . Further, this limit resides in the regime of large where conventional methods fail (Figure 2). We further consider the limit where the intersection between any pair of the subspaces in is null. In this limit, the marginal distribution of all of the data in any one of the can be appear unimodal due to a dominance of data points from the clusters for which this subspace is irrelevant, despite data in the clusters ,
showing a bimodal signature in this subspace. Hence finding the identity of the informative subspaces by distinguishing moments of the marginal distribution is not possible as
grows or data density decreases, even in the case of normally distributed data (Figure 2, 3B). In this limit, the values of
corresponding to informative dimensions are , and 0 for uninformative dimensions. Our reason for studying this limit of pairwise separability is that an algorithm that can find the informative subspaces of in these limits should be able to do so in instances where in the dimensions of are larger than one and intersecting.We generate data such that the mean of the marginal distributions of clusters and along a specific whose span defines are separated by and the sample variance of each cluster’s marginal distribution is . The marginal distribution of cluster in all other dimensions, i.e. , is unimodal with zero mean and unit variance. Therefore, in all there are dimensions in each of which which a pair of clusters are linearly separable, while the other clusters are not, and dimensions where all clusters are drawn from the same unimodal distribution. Normalizing each feature to have unit variance leaves one free parameter, , which controls the pairwise separability of clusters within their informative subspace (Figure 3B).
We develop an approach to estimate the weight vector knowing neither the identity of points belonging to each cluster nor the total number of clusters. In order to estimate , we propose to average estimates of over an ensemble of clustering configurations. Specifically, we sample an ensemble of possible clustering geometries, , from each of which a collection of max-margin classifiers are computed in order to compute using Equation 1:
(2) |
Where
is the probability of a clustering configuration given the data. This sum can be approximated numerically through a sampling procedure outlined in Algorithm 1, where cluster proposals are sampled according to
(3) |
where, is the number of clusters, and is our prior over the number of proposal clusters.
Consider one such proposed clustering configuration with clusters, denoted by where each indexes the data points that belong to the th proposed cluster. For this proposed clustering configuration, we compute a set of classifiers that separate each pair of clusters. Based on the assumption that is sparse, or equivalently that the true are low dimensional, we impose an L1-regularized max margin classifier to compute from the data X and the proposed cluster configuration as in [13]:
(4) |
Where indicates the positive component, and is a sparsity parameter. We set such that the expected number of non-zero components in each is 1. Specifically, we sample cluster configurations by clustering on random subsets of the data, and average the weights of max-margin classifiers over this ensemble:
(5) |
This procedure is outlined explicitly in Algorithm 1.
In the challenging regime of large , this algorithm can robustly identify key features in the data. In particular, as increases, there is a scaling of the algorithms performance as a function of , as well as a dependence on the number of data points . First, we sample a variety of proposal clusters , each with clusters drawn from a prior . Using counting arguments (see 4.2) we can estimate the frequency of proposed aligning with informative with a bimodal signature and uninformative features without. This ratio of the average weights of informative dimensions to the average of the uninformative dimensions, , scales as . The scaling, however, also depends on data density. Specifically, consider the length scale separating two neighboring data points in the full space scales as . In the relevant subspace , this length scale translates to a volume of which must be compared to the characteristic volumes in this subspace that reflect the multi-modal structure of the data. If the identities of the true clusters in are known, one can ask what the errors are in clustering in the full space instead of in by computing the entropy, of the composition of inferred clusters based on the true cluster identities of data points. This entropy has to be a function of the ratio of the characteristic volumes in to . Or equivalently, the entropy of the clusters should be a monotonically increasing function , denoting increasing errors in clustering. The form of the function depends on the true data distribution and the clustering method. Therefore, our expectation for the ratio of counts for the informative dimensions and counts for the uninformative dimensions should scale like
(6) |
We numerically generated Gaussian distributed data for
, using , , and ran 2000 iterations of the algorithm with and found close agreement for the range of parameters (Figure 4A,4B).In order to estimate the significance of frequencies produced by the algorithm, we constructed a null models to estimate in the absence of signal. First, each column of the data matrix is shuffled to produce a null distribution . This leaves marginal distributions of each dimension unchanged. Next, each feature can be scored based on the null distribution, , and statistics , and
can be computed from these scores. We then can compute a Z-score for each dimension as
. Motivated by the work of [14] and [15], more precise estimates could be obtained by generating synthetic marginals without multimodality, which is an area for future work.Correlations in the uninformative subspace can lead to erroneous counts in the correlated axes. This is caused by correlations in uninformative dimensions biasing the proposal clusters to be differentially localized in these axes. Despite these false positives, the false negative rate remains low, resulting in minimal degradation of the ROC curve (Figure 4C). In practice, eliminating any number of uninformative dimensions is effective in restricting analysis to a smaller regime of . Thus, even in the presence of false positives, removing uninformative dimensions prior to conventional analysis can increase the accuracy of clustering or dimensionality reduction techniques.
A free parameter in the synthetic data generated in Figure 4 is , the ratio of mean separation to variance of distributions in the informative subspace, which controls the separability of clusters. As decreases, we see degradation in the AUROC for our algorithm, but identification of key dimensions is still possible even as the mean separation approaches the noise level in the distributions (Figure 4D, inset).
A central challenge in developmental biology is the characterization of cell types that arise during the course of development, and an understanding of the genes which define and control the identity of cells as they transition between states. Starting at fertilization, embryonic cells undergo rapid proliferation and growth [6, 16]. In a mouse, these cells form the epiblast, a cup shaped tissue surrounded by extra-embryonic cells by E6, or 6 days after fertilization. Only the cells of the epiblast will go on to give rise to all the cells of the mouse. These cells are pluripotent, meaning they have the developmental potential to become any cell type in the adult mouse body [17]. At E6, proximal and distal sub-populations of both the epiblast and surround extra-embryonic cell types begin secreting signaling proteins [18]
, which when detected by nearby cells, can increase or decrease the expression of transcription factors – proteins that modulate gene expression, and can thus change the overall expression profile of a cell. Signaling factors direct genetic programs within cells to restrict their lineage potential and undergo transcriptional as well as physical changes. Posterior - proximal epiblast cells migrate towards the outside of the embryo forming a population called the primitive streak, in a process called gastrulation which takes place between E6.5 and E8. This time frame is notably marked by the emergence of three populations of specified progenitors known as the germ layers
[19]: endoderm cells, which later differentiate into the gastrointestinal tract and connected organs, mesoderm cells, which have the potential to form internal organs such as the muscoskeletal system, the heart, and hematopoietic system, and ectoderm cells, which later form the skin and nervous system. The mesoderm can be subdivided into the intermediate mesoderm, paraxial mesoderm, and lateral plate mesoderm, which each have further restricted lineage potential. Identifying the key transcription factors that define and control the genetic programs that lead to these distinct subpopulations will allow for experimental interrogation and a greater understanding of the gene regulatory networks which control development.Recent advances in single cell RNA-sequencing technology allow for simultaneous measurement of tens of thousands of genes [8, 9] during multiple time points during development. These technological advances promise to provide insight into the identity and dynamics of key genes that guide the developmental process, yet even clustering cells into types of distinct developmental potential, and identifying the genes responsible for the diversity has been difficult [20, 21] . Existing methods typically find signal in correlations between large numbers of genes with large coefficients of variation to determine a cell’s states. However, experimental evidence suggests that perturbations of a small number of transcription factors are sufficient to alter a cell’s developmental state and trajectory [5, 6, 7]. Further, recent work suggests that a small set of four to five key transcription factors is sufficient to encode each lineage decision [11, 22]. We therefore believe that signature of structure in these data resides in a low dimensional subspace. While many existing methods rely on hand-picking known transcription factors responsible for developmental transitions [12], we attempt to discover a low dimensional subspace of gene expression which encodes multi-modal expression patterns indicating the existence of distinct cell states.
In [12]
, single-cells are collected from a mouse embryo between E6.5 and E8.5, encompassing the entirety of gastrulation, and profiled with RNA-sequencing to quantify RNA transcriptional abundance. We considered 48692 cells from E6.5 - E7.75 which had more than 10,000 reads mapped to them. We then sub-sampled reads such that each cell had 10,000 reads. Individual genes were removed from analysis if they had a mean value of less than 0.05, or a standard deviation of less than 0.05 (based on
[20, 21]). We restricted our analysis to transcription factors because, as regulators of other genes, variation in transcription factor expression is a strong indication of biological diversity between cells, or cell types. We normalized the 409 transcription factors with expression above these thresholds to have unit variance. A cell-cell correlation analysis, followed by hierarchical clustering fails to capture the fine grained diversity of cell types that is known to exist at this time point (Fig 5A).
We attempted to discover a low dimensional subspace in which signatures of cell type diversity could be inferred using the algorithm outlined in the previous section. We sampled 3000 clustering configurations based on hierarchical (ward) clustering of 5000 subsampled cells, with with , chosen to cover a range around the 37 clusters found in [12]. We find 27 transcription factors with a z-score , 18 of which have known have previously identified essential functions in the regulation of differentiation during gastrulation (See Table 1).
Our hypothesis is that the variation in the 27 discovered transcription factors provides a subspace in which multimodal signatures allow the identification of cell types. However, single cell measurements of individual genes are known to be subject to a variety of sources of technical noise [23]. In order to decrease reliance on individual measurements, we take each of the 27 transcription factors with high scores, and extend the subspace to include 5 genes (potentially not transcription factors) that have the highest correlation with each of the 27 discovered transcription factors, resulting in an expanded subspace of 83 genes in which to cluster the data (Full list in supplement). The cell-cell covariance matrix in this subspace (Figure 5B), reveals distinct cell types and subtypes, and a heat map of the expression levels of these 83 genes shows differential expression between subtypes of cells.
We hierarchically clustered the cells into 35 cell types based on expression of these 83 genes. The corresponding identity of these cell types was determined using the expression pattern of all genes (Table 2, Figure 5c), and identify extraembryonic populations (C5-9,C18-20), epiblast populations (C27-C34), primitive streak populations (C14-17,C23-25), mesoderm subtypes (C2-4,C12,C13,C16,C17,C22), endoderm (C21), and primitive erythrocyte (C1). For example, we find a subpopulation of epiblast cells that have upregulated Nanog (as well as other early markers of the primitive streak), suggesting that these cells are positioned on the posterior-proximal end of the epiblast cup [24]. The large primitive streak population, which extends along the proximal side of the embryo, contains subtypes distinguished by Gsc [25] and Mesp1 [26], which give rise to distinct fates. We find a distinct population of anterior visceral endoderm cells, marked by Otx2 and Hhex, which define the population responsible for the anterior-posterior body axis [27]. This population, which is distinguished from other Foxa2-expressing subpopulations of the visceral endoderm, is crucial for proper development.
Most importantly, in extracting the relevant features from the data, our algorithm identifies known and validated transcription factors that are crucial to the developmental processes happening in this time frame. Further, by eliminating extraneous measurements, we are able to identify clear differential expression patterns between sub-types of cells which were indistinguishable through previous methods. In particular, identification of primitive streak subpopulations provides novel insight into a central developmental process, and we identify key genes that would allow for experimental interrogation of the spatial organization of the sub-types and their dynamics.
Table 1 | |||
Gene Name | Associated Cell Type | Citation | |
Creb3l3 | 34.20 | ||
Tfeb | 13.37 | ||
Rhox6 | 10.15 | ||
Elf5 | 9.45 | Extraembryonic Ectoderm | [28] |
Gata1 | 8.77 | Primitive Erythrocite | [29] |
Pou5f1 | 8.38 | Epiblast, Primitive Streak | [24] |
Sox17 | 5.61 | Endoderm | [30] |
Nr0b1 | 5.58 | ||
Hoxb1 | 4.54 | Mesoderm | [31] |
Foxf1 | 3.36 | Lateral Plate Mesoderm | [32] |
Gata2 | 3.27 | Extraembryonic Mesoderm | [33] |
Prdm6 | 3.12 | ||
Bcl11a | 2.99 | ||
Foxa2 | 2.70 | Anterior Visceral endoderm, Anterior primitive streak | [27], [34] |
Gsc | 2.67 | Anterior Primitive Streak | [25] |
Hand1 | 2.57 | Posterior Mesoderm, Lateral Plate Mesoderm | [35] |
Ascl2 | 2.52 | Ectoplacental Cone | [36] |
Mesp1 | 2.46 | Posterior Primitive Streak | [26] |
Hoxa1 | 2.44 | Mesoderm | [31] |
Nanog | 2.24 | Epiblast | [24] |
Zfp42 | 2.14 | Extraembryonic Ectoderm | [37] |
Cdx1 | 2.12 | Paraxial Mesoderm | [38] |
Runx1 | 1.82 | ||
Hoxb2 | 1.73 | Mesoderm | [31] |
Id2 | 1.51 | Extraembryonic Ectoderm | [39] |
Tbx3 | 1.31 | ||
Pitx2 | 1.20 | ||
Table 2 | ||
---|---|---|
Cluster ID | Label | Markers |
0 | Unclustered | |
1 | Primitive Erythrocyte Progenitor [29] | Hba-x [40], Hbb-bh1 [41], Gata1 [29], Lmo2 [42] |
2 | Mesoderm | Car3, Spag5, Hoxb1 [31] , Cnksr3, Smad4, Zfp280d, Vim [43], Ifitm1 |
3, 4 | Lateral Plate Mesoderm | Foxf1 [32], Hand1 [35] |
5 | Anterior Visceral Endoderm | Sox17, Foxa2 [27], Cer1 [44], Frat2, Lhx1 [45], Hhex [46], Gata6, Ovol2, Otx2 [27], Sfrp1 [47] |
6,7 | Extraembryonic Mesoderm | Fgf3 [48], Lmo2 [42], Gata2 [33], Bmp4 [49] |
8,9 | Visceral Endoderm 1 | Rhox5 [50], Emb [51], Afp [52] |
10,11 | Neuromesodermal Progenitor | Sox2, T [53] |
12,13 | Paraxial Mesoderm / Presomitic Mesoderm | Hoxa1, Hoxb1 [31], Cdx1, Cdx2 [38] |
14 | Posterior Primitive Streak | Mesp1 [26], Snai1 [54], Lhx1 [45, 55], Smad1 [56] |
15 | Anterior Primitive Streak, Organizer-like Cells | Foxa2 [34], Gsc [25], Eomes [34] |
16,17 | Posterior Primitive Streak derived Mesoderm, Lateral Plate Mesoderm Progenitors | Msx2 [57], Snai1 [54], Foxf1 [32], Hand1 [35], Gata4 [58] |
18,19 | Extraembryonic Ectoderm | Cdx2 [59], Rhox5 [50], Id2 [39], Gjb5 [60], Tfap2c [28], Zfp42(aka Rex1) [37], Elf5 [28], Gjb3 [60], Ets2 [61] |
20 | Ectoplacental Cone | Plac1 [61], Ascl2 [36] |
21 | Definitive Endoderm | Sox17 [30], Foxa2 [62], Apela [63] |
22 | Mesendo Progenitor, Primitive Streak | Tcf15 [64], Cer1, Hhex [65] |
23 | Posterior Primitive Streak, Cardiac Mesoderm Progenitors | Mesp1 [26], Gata4 [58], Lhx1 [55], Smad1 [56] |
24,25 | Primitive Streak | T, Mixl1, Eomes, Fgf8, Wnt3 |
26 | Klf10, Gpbp1l1, Hmg20a, Rbm15b, Celf2 | |
27-28 | Posterior-Proximal Epiblast | Nanog [24], Sox2 [66], Pou5f1 [24], Otx2 [67] |
29-34 | Epiblast | Sox2 [66], Pou5f1 [24], Otx2 [67] |
Various methods exist to select features for clustering. Some are based on correlation analysis in the full space [68], which we have shown only be effect in the regime of small . Our approach is based off of constructing max-margin classifiers which do not rely on generative models of the data distribution, and does not attempt to exactly infer the clustering configuration. This is in general an advantage because it does not rely on context-specific knowledge of the underlying process. For a full discussion of the comparison, see [69]
. However, the identification of a small subset of informative features can be formulated as a Bayesian inference problem, where a log likelihood function is maximized over the hidden parameters via an expectation maximization scheme
[70]. In fact, model based clustering has been explored in depth in [71, 72], and adapted to feature selection by the inclusion of a lasso term on the separation of the first moments in
[73, 74, 75]. Advantages and drawbacks of these are discussed in [2], which provides a more general framework. In [2], a feature weight vector is introduced and learned by amending a clustering cost function with a penalty on the feature weights. In particular, their method can be used to adapt the K-Means or hierarchical clustering cost function to include a sparsity constraint on feature weights, which is learned simultaneously with the clustering configuration during training via an iterative optimization. Their approach successfully adapts K-means and hierarchical clustering to perform better in the large regime, however there are two main drawbacks. 1) For the K-means variation, estimation of cluster number is reliant on the noisy and unpredictable gap statistic (see for example, the discussion in [76] ), and estimating the correct number of clusters is challenging. When the number of clusters in not known, informative features are frequently missed (Figure S2). 2) The sparse hierarchical clustering in [2] was repeated 5 times on the mouse data described in the previous section and did not converge after 2 days. We then subsampled the cells from 48692 to 5000 and ran 5 iterations on these subsampled data sets, none of which converged after 2 days. Our method is distinguished by integrating out the cluster configuration to provide more general identification of key features, and providing a faster approximation of key features on large data sets.Identifying sub-spaces which define classes and states from high dimensional data is an emerging problem in scientific data analysis where an increasing number of measurements push the limits of conventional statistical methods. Techniques such as PCA and ICA provide invaluable insight in data analysis, but can miss multimodal features, particularly in high dimensional settings. These methods which have reduced success in the regime can be supplemented by our technique by finding a lower dimensional subspace in which further analysis can be conducted. Crucially, eliminating any informative dimensions decreases the ratio, moving to a regime in which conventional methods are more effective. By reducing the dimensionality of the data, it is possible to artificially increase data density, and mitigate associated problems that are prevalent in high dimensional inference. Further, as our algorithm can be a wrapper over any clustering algorithm to construct the proposal clusters, it has varied applicability in settings where K-means or other specific clustering algorithms are unsuccessful.
Biological data from neural recordings, behavioral studies, or gene expression is increasingly high dimensional. Identifying the underlying constituents of the system that define distinct states is crucial in each setting. In contexts such as transcriptional analysis in developmental biology, finding the key genes that define cell states is a central problem that bridges the gap between high throughput measurements and mechanistic experimental follow ups. Identification of transcription factors with multimodal expression that define cellular states allows for the study of dynamics of state transitions and spatial patterning of the embryo. Our method rediscovers known factors in well studied developmental processes and predicts several gene candidates for further study. Identifying defining features in high dimensional data is a crucial step in understanding and experimentally perturbing systems in a range of biological domains.
We would like to thank Deniz Askel for extensive help with annotating the clusters. In addition we would like to thank Anirvan Sengupta, Massimo Vergasola, Sam Kou, Matt Thomson, Gautam Reddy, Sean Eddy, and members of the Ramanathan lab for discussions and comments on the manuscript. The work was supported by DARPA W911NF-19-2-0018 , 1R01GM131105-01 and DMS-1764269
In situations where some dimensions or features do not carry relevant information For data points , if is the subspace with meaningful information, and is just noise.
We are first interested in distances between . Let . We first want to compute the correlation between and . If and then:
By definition, , and so
The signal in and is uncorrelated, so the second term is . are both differences of multivariate Gaussians, which are also multivariate Gaussian. And if we let then is also multivariate Gaussian. We want to compute . For a multivariate Gaussian , ,
So
The trace is the sum of the eigenvalues squared, so if the data is normalized such that the eigenvalues are of order 1, this scales like
Let be the number of proposed clusters per true cluster (). We assume that the entropy relative to the true class labels is low in each of the proposed clusters (in practice, this condition is met in situations where ). This is true as long as the prior on the proposal cluster number is non-zero for , because the expectation of in Equation 5 is dominated by larger values of because each proposal contributes terms to the sum. We start by counting the number of pairs of proposed clusters wherein cluster in the pair contains data points primarily belonging to the same true cluster:
And so the frequencies of errors is given by
If the true clusters have roughly equal sizes, then we can assume that for all , and this ratio reduces to
The signal frequency is given by a count for each feature when the two proposal clusters in a pair contain points primarily from different true clusters, of which there are roughly
These votes all identify an informative feature, so the fraction is given by
When we assume each , this reduces to:
The ratio of tells us the regime in which the noise is much lower than the signal for meaningful features:
Here are the number of uninformative features, and is the number of informative features, so this scales like
Single cell RNA-seq data from mouse gastrulation [12]: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE87038
1normm support vector machines.
In Proceedings of the 16th International Conference on Neural Information Processing Systems, NIPS’03, pages 49–56, Cambridge, MA, USA, 2003. MIT Press.On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes.
Adv. Neural Inf. Process. Sys, 2, 04 2002.