1 Introduction
Finding low dimensional embeddings of high dimensional data is vital in understanding the organization of unsupervised data sets. However, most embedding techniques rely on the assumption that the data set is locally Euclidean
[7, 15, 1]. In the case that features carry implicit weighting, some features are possibly irrelevant, and most points are missing some subset of the features, Euclidean neighborhoods can become spurious and lead to poor low dimensional representations.In this paper, we develop the method of expert driven functional discovery to deal with the issue of spurious neighborhoods in data sets with high dimensional contrasting features. This allows small amounts of input and ranking from experts to propagate through the data set in a non linear, smooth fashion. We then build a distance metric based off these opinions that learns the invariant and irrelevant features from this expert driven function. Finally, we locally normalize this distance metric to generate a global embedding of the data into a homogeneous space.
An example to keep in mind throughout the paper, an idea we expand upon in Section 4, is a data set containing publiclyreported measurements of hospital quality. The Center for Medicare and Medicaid Services Hospital Quality Initiative reports approximately 100 different measures describing various components of the quality of care provided at MedicareCertified hospitals across the United States. These features range in measuring hospital processes, patient experience, safety, rates of surgical complications, and rates of various types of readmission and mortality. There are more than hospitals that reported at least on measure during 2014, but only hospitals with 90% measures reported. The measures are computed quarterly, and are publicly available through the Hospital Compare website [8]. The high dimensional nature of these varied measures make comprehensive inferences about hospital quality impossible without summarizing statistics.
Our goal is more than just learning a ranking function on the set of hospitals . We are trying to characterize the cohort of hospitals and organize the geometry of the data set, and learn a multidimensional embedding of the data for which the ranking function is smooth. This gives an understanding of the data that doesn’t exist with a one dimensional ranking function. Specifically, we are looking for metafeatures of the data in order to build a metric that induces a small Lipschitz constant on the function , as well as on features measured by CMS.
An example of this organization is shown in Figure 1. The organization is generated via our algorithm of expert driven functional discovery, the details of which are found in Sections 2 and 3. The colors in each image correspond to three notable CMS features: risk standardized 30 day hospitalwide readmission, patient overall rating of the hospital, and risk standardized 30 day mortality for heart failure. This organization successfully separates hospitals into regimes for which each feature is relatively smooth.
Our organization is accomplished via a two step processing of the data. First, we look to characterize the topology of the data set by creating a stable representation of such that neighborhoods can be differentiated at varying scales, and neighborhoods are preserved across varying parameters of the algorithm. This is done in Sections 2 and 3, using a spin cycling of deep nets trained on some expert driven function. Second, we build a metic on that topology by taking a local Mahalanobis distance on the stable neighborhoods (i.e. local zscoring) [11, 16]. This is done in Section 3.3, and guarantees that the induced metric is homogeneous. This means that, if denotes the neighborhood of a point , then for measures the same notion of distance as for .
Figure 2 demonstrates the issue we refer to here. The left image shows an organization of the hospitals in which neighborhoods have not been normalized. The red points refer to “good” hospitals, and the blue refer to “bad” hospitals (in a naive sense). The organization on the left gives no notion of the spread of the points and leaves certain questions unanswerable (i.e. are “good” hospitals as diverse as “bad” hospitals). However, if we locally zscore the regions of the hospitals, as in the right figure, it the global notion of distance is normalized and it becomes clear that “good” hospitals share many more similarities amongst themselves than “bad” hospitals do among themselves.
Also, by taking a local zscoring of the features, we generate an organization that is dependent only on the neighborhoods , rather than being dependent on the specific representations used. Figure 3 shows the organizations of the hospitals generated by algorithms with two very different parameter sets which, after zscoring the neighborhoods, generate similar embeddings. More details about this embedding can be found in Section 4.3.3.
However, discovering the topology of the hospitals is nontrivial. The features may have significant disagreement, and not be strongly correlated across the population. To examine these relationships, one can consider linear correlations via principal component analysis. The eigenvalues of the correlation matrix do not show the characteristic drop off shown in linear low dimensional data sets. In fact, 76 of the 86 eigenvalues are above
the size of the largest eigenvalue. Previous medical literature has also detailed the fact that many of the features don’t always correlate [10, 3].For this reason, there does not exist an organization for which all features are smooth and monotonically increasing. This is why the metafeatures, and organization, must be driven by minimal external expert opinion. This observation makes the goal of our approach three fold: develop an organization of the data that is smooth with respect to as many features as possible, build a ranking function that agrees with this organization, and minimize the amount of external input necessary to drive the system.
Our expert driven functional discovery algorithm blends the data set organization of diffusion maps and coupled partition trees with the rich set of nonlinear features generated by deep learning and stacked neural nets. We build an initial organization of the data via coupled partition trees
[5], and use this partitioning to generate pseudopoints that accurately represent the span of the data at a coarse scale. This step is explained in Section 2. This organization is analyzed and used as an input to a series of stacked neural nets, which learn invariant representations of the data and separate disparate clusters of points. This step is explained in Section 3. See [2] for a review of stacked neural nets and deep learning.It is important to note that our use of stacked neural nets is different from traditional deep learning applications. We discuss these differences in Section 3.5. The purpose of using deep learning and organizing of the generated representations is to create a notion of fine neighborhoods between points; neighborhoods where the number of neighbors scales smoothly with the distance metric.
We then examine and validate our algorithm on the CMS Quality Initiative features in Section 4.
2 Information Organization and Expert Driven Functional Discovery
2.1 Training on Data with Full Knowledge
Let the data matrix be
(1) 
where
is a vector of observations describing the
data point. Each is allowed to have arbitrarily many missing entries. Define supp.Due to the missing entries, calculating an affinity between every two points and is not necessarily possible, given that the intersection of the supports of their known values may be small or even disjoint. For this reason, we begin by restricting ourselves to points that have at most missing entries. We shall begin by organizing the set of points .
To gain an initial understanding of the geometry of
, we consider the cosine affinity matrix
where(2) 
where the inner product is calculated only on the entries in suppsupp. By definition of , this set contains at least known values.
The cosine affinity matrix serves as a good starting point for learning the geometry of . However, we must develop a way to extend any analysis to the full data . For this reason, we partition both the data points and the observation sets of . This gives us two advantages: partitioning the data points captures the ways in which different observations may respond to different subsets of , and partitioning the observation sets into similar question groups gives a method for filling in the missing observations in .
We construct a coupled geometry of using the algorithm developed in [5]. The initial affinity is given by the cosine affinity matrix , and the iterative procedure is updated using Earth Mover Distance [14].
Remark: Let the final affinity matrix be called . Let the eigenpairs of be called with . Then the organization of is generated by
where is the dimension of the underlying manifold.
2.2 Filling in Missing Features
Let be the hierarchical tree developed on the observations in from Section 2.1. Let the levels be , with the nodes for level named . Let be a data point with the entry missing. In order to add into the geometry of
, we must estimate the entries in
to calculate an affinity between and other points.gives a tree of correlations between the observations. This allows us to fill in with similar, known entries. Find the lowest level of the tree (most strongly correlated questions) for which observation and such that is known. Then the estimate of satisfies
(3) 
Along with an estimate of , (3) also gives a level of uncertainty for the estimate, as smaller (i.e. coarser folders) have lower correlation and give larger reconstruction error.
2.3 Expert Driven Function on the Folders
Let be the hierarchical tree developed on the data points in from Section 2.1. Let the levels be , with the nodes for level named . As the partitioning becomes finer (i.e. approaches ), the folders contain more tightly clustered points. This means that the distance from an point to a centroid of a folders becomes smaller as the partitioning becomes finer.
Fix the level in the tree. The centroids of these folders can be thought of as “types” of data points, or pseudopoints. There are two major benefits: there are a small number of pseudopoints relative to that span the entire data space, and the pseudopoints are less noisy and more robust to erroneous observations.
These pseudopoints are the key to incorporating expert knowledge and opinion. The pseudopoints are easier and much faster to classify than individual points, as there are a small number and they are less noisy than individual points. Also, the pseudopoints effectively synthesize the aggregate performance of multiple hospitals. The classifications generated by experts can be varied, anything from quality rankings to discrete classes to several descriptive features or “metafeatures” of the bins. Specifically, the user assigns a set of classes
and a classification function such that(4) 
This function is understood as a rough estimate, since the classification is applied to all even though the class is determined only from the centroid of .
This step is nontraditional in unsupervised classification algorithms. Traditionally, the clustering is done agnostic to the final goal of classification. However, this does not allow for a second step correction of the original classification. By allowing a user driven component that quickly examines the cluster centroids, we are able to learn an initial classification map for the points. This provides a rough characterization of which clusters should be collapsed due to similar classification scores, which clusters should be separated further due to drastically different classification scores, and a method of determining class labels for points on the border of multiple clusters.
This function gives a rough metric that has dependencies of the form
(5) 
This metric needs to satisfy two main properties:

such that if for and some norm , and

if .
A metric that satisfies these two properties naturally relearns the most important features for preserving clusters while simultaneously incorporating expert knowledge to collapses nonrelevant features. We learn this function using neural nets, as described in Section 3.
3 Deep Learning to Form MetaFeatures
There are entire classes of functions that approximate the behavior of in (5
). For our algorithm, we use neural nets for several reasons. First, the weight vectors on the first layer of a neural net have clear, physical interpretation, as the weight matrix can be thought of as a nonlinear version of the weight vectors from principal component analysis. Second, current literature on neural nets suggest a need for incredibly large datasets to develop meaningful features on unsupervised data. Our algorithm provides a way to turn an unlabeled data set into a semisupervised algorithm, and incorporates this supervision into the nodes of the neural net. This supervision appears to reduce the number of training points necessary to generate a nontrivial organization. Past literature has used radial basis functions as the nonlinear activation for deep learning
[4] , as well as for analysis of the structure of deep net representations [12].3.1 Neural Nets with BackPropagation of Rankings
For our algorithm, we build a 2 layer stacked autoencoder with a sigmoid activation function, with a regression layer on top. The hidden layers are defined as
with
being a sigmoid function applied elementwise, and
. The output functionis logistic regression of these activation units
The reconstruction cost function for training our net is an reconstruction error
(6) 
The overall loss function we minimize, which combines the reconstruction cost with several bounds on the weights, is
Note that we rescale if it takes values outside
. We then backpropagate the error by calculating
and and adjusting the weights and bias accordingly. See [13] for a full description of the algorithm.Definition 3.1.
The deep neural net metric on with respect to an external function is defined as
Lemma 3.2.
A deep neural net with a logistic regression on top generates a metric that satisfies Condition 1 from (5) with a Lipschitz constant of with respect to Euclidean distance. The output function also has a Lipschitz constant of with respect to Euclidean distance.
Proof.
Let . Then . By the mean value theorem, . Then
The same argument applies for .
∎
Lemma 3.3.
A two layer neural net with a logistic regression on top creates a function which satisfies a variant of Condition 2 from (5), namely that
(7) 
where , , and is the expected value over the set .
Moreover, the deep neural net generated also satisfies
(8) 
Proof.
We have
Unfortunately, because (6) is a global minimization, we cannot say anything meaningful about the difference for individual points. However, we do have
where denotes the number of elements in this set. Let and . Then
This means that by minimizing , we are forcing the separation of points with different initial ranking to be as large as possible. This makes enforces Condition 2 of (5) in the aggregate over all such points.
The scaling of for is a simple application of Lemma 3.2. ∎
3.2 Heat Kernel Defined by
The weights generated by the neural net represent “metafeatures” formed from the features on . Each hidden node generates important linear and nonlinear combinations of the data, and contains much richer information than a single question or average over a few questions.
One downside of traditional deep learning is that neural nets only account for the geometry of the observations. They ignore the geometry and clustering of the data points, opting to treat every point equally by using a simple mean of all points in the cost function (6
). This can easily miss outliers and be heavily influenced by large clusters, especially with a small number of training points.
The questionnaire from Section 2.1 organizes both the observations and the data points, though the metafeatures from the questionnaire are simply averages of similar features as in (3). However, the expert ranking assigned to each bin reflects a rough initial geometry onto the points to be learned by the neural net.
The back propagation of the expert driven function is essential to building significant weight vectors in this regime of small datasets. When the ratio of the number of data points to the dimension of the features is relatively small, there are not enough training points to learn the connections between all the features without considering the points themselves. The back propagation of the classification function generated from the questionnaire is a way to enforce the initial geometry of the data on the weight vectors. The hospital ranking example in Section 4 demonstrates the need for back propagation, and Figure 7 demonstrates the fact that the features from a simple SDAE are not sufficient for separating data points.
For our algorithm, the SDAE is trained on , the subset of points with “full” collection of features. This is to avoid training on reconstructed features which are subject to reconstruction error from Section 2.2.
Another problem with neural nets is that they can be highly unstable under parameter selection (or even random initialization). Two identical iterations can lead to completely different weights set. Along with that, back propagation can force points into isolated corners of the cube in .
For this reason, we rerun the neural net times with varied random seeds, number of hidden layers, sparsity parameters, and dropout percentages. After iterations, we build the new set of features on points as . This defines an adjacency matrix on with affinity defined between two points as
(9) 
Along with that, the final ranking function on comes from . Note that
For each neural net , we keep the number of hidden layers small relative to the dimension of the data. This keeps the net from overfitting the data to the initial organization function .
3.3 Standardizing Distances to Build an Intrinsic Embedding
While this generates a global embedding based off local geometry, it does not necessarily generate a homogenous space. In other words, does not necessarily guarantee that and differ by same amount as and . This is because and may differ in a large number of deep net features, whereas and may only differ in one or two features (though those features may be incredibly important for differentiation).
For this reason, we must consider a local zscore of the regions of the data. For each point , there exists a mean and covariance matrix within a neighborhood about such that
This generates a new whitened distance metric
(10) 
where is the PenroseMoore pseudoinverse of the covariance matrix.
One can generate a final, locally standardized representation of the data via the diffusion kernel
and the low frequency eigenvalues/eigenvectors of , which we call . The final representation is denoted
(11) 
3.4 Extension to New Points
Once metafeatures have been built, extending the embedding to the rest of can be done via an asymmetric affinity kernel, as described in [9]. For each , one can calculate and with the weights generated in Section 3.2. This defines an affinity between and via
The affinity matrix is . [9] then shows that the extension of the eigenvectors of from (9) to the rest of is given by
where is the row stochastic normalization of .
Similarly, once the embedding is extended to all points in , we can extend the final, standardized representation to the rest of as well using the affinity matrix
for and . Once again, the eigenvectors in (11) can be extended similarly via
where is the row stochastic normalized version of .
In this way, can be defined on all points after analysis on the reference set .
3.5 Different Approach to Deep Learning
Our algorithm, as we will discuss in detail in Section 3, uses the metafeatures from stacked neural nets in a way not commonly considered in literature. Most algorithms use back propagation of a function to finetune weights matrices and improve the accuracy of the one dimensional classification function. However, in our algorithm, the purpose of the back propagation is not to improve classification accuracy, but instead to organize the data in such a way that is smooth relative to the classification function. In fact, we are most interested in the level sets of the classification function and understanding the organization of these level sets.
At the same time, we are not building an autoencoder that pools redundant and correlated features in an attempt to build an accurate, lossy compression (or expansion) of the data. Due to the high level of disagreement among the features, nontrivial features generated from an autoencoder are effectively noise, as we see in Section 4.2. This is the motivation behind propagating an external notion of “quality”.
4 Expert Driven Embeddings for Hospital Rankings
4.1 Hospital Quality Ranking
For preprocessing, every feature is mean centered and standard deviation normalized. Also, some of the features are posed in such a way that low scores are good, and others posed such that low scores are bad. For this reason, we “depolarize” the features using principal component analysis. Let
be the data matrix. Then let be the largest eigenvector of the covariance matrixand take the half of the features for which . This makes scores above the mean “good” regardless of whether the feature is posed as a positive or negative question.
We begin by building a questionnaire on the hospitals. Our analysis focuses on the 2014 CMS measures.We put a weight on mortality features and
weight on readmission features due to importance, because the outcome measures describe the tangible results of a hospitalization are particularly important for patients. The questionnaire learns the relationships between the hospitals, as well as the relationships between the different features. In doing this, we are able to build a partition tree of the hospitals, in which hospitals in the same node of the tree are more similar than hospitals in different nodes on the same level of the tree. As a side note, the weighting on mortality features only guarantees that the intrabin variance of the mortality features is fairly low.
For the ranking in this example, we use the layer of a dyadic questionnaire tree, which gives bins and pseudohospitals. Experts on CMS quality measures rank these pseudohospitals on multiple criteria and assign a quality score between to to each pseudohospitals. We use an average of nodes per layer of the neural net. Also, we average the results of the neural nets over trials. We shall refer to the final averaged ranking as the deep neural net ranking (DNN ranking) to avoid confusion.
4.2 Two Step Embedding of Hospitals
Figure 4 shows the embedding of the subset of hospitals used in training the neural net. It is colored by the quality function assigned to those hospitals. The various prongs of the embedding are explained in Section 4.3.4. Figure 5 shows the embedding of the full set of hospitals. This was generated via the algorithm in Section 3.4.
Finally, while the goal of the hospital organization is to determine neighborhoods of similar hospitals, it is also necessary for all hospitals in a shared neighborhood to share a common quality rating. Figure 6 plots the quality function assigned to the hospitals against the weighted average quality function of its neighborhood, where the weights come from the normalized affinities between the given hospitals and its neighbors. The strong collinearity demonstrates that the assigned quality function is consistent within neighborhoods of similar hospitals.
To demonstrate that the expert input back propagation is necessary for a viable ranking and affinity, we include Figure 7. Here, we build the same diffusion map embedding, but on the features of the autoencoder before back propagation of the expert input function. Due to the small number of data points relative to the number of features, an untuned autoencoder fails to form relevant metafeatures for the hospitals.
4.3 Internal Validation
The problem with validating this composite rating of hospital quality is the lack of external ground truth. For this reason, we must rely on internal validation mechanisms to demonstrate consistency. There are two qualities necessary for a ranking mechanism: the affinity between hospitals cannot disagree too drastically with the original features, and similar hospitals must be of similar quality.
For the rest of the section, we refer to several simple metafeatures for comparison and validation. Row sum ranking refers to a simple ranking function
where are the normalized and depolarized features for a given hospital . Average process, survey, and outcome features are the same as the row sum score, but restricted only to features in the given category. NNLS weighted ranking refers to a function
where the weights are calculated using nonnegative least squares across the normalized and depolarized features, with the DNN ranking as the dependent variable. NNLS weighted process, survey, and outcome features are the same as the NNLS weighted score, but restricted only to features in the given category. NNLS was used for calculating the weights to avoid overfitting the DNN ranking by using negative weights.
Also, our focus (unless otherwise stated) is on the subset of hospitals that have more than features reported. This is because a number of our validation steps use the row sum ranking and NNLS weighted ranking, which are more accurate when calculated over nonmissing entries.
4.3.1 DNN Satisfies Conditions for
Figure 8 verifies that the back propagation neural net satisfies Condition 1 of from (5). The plot shows the ranking of each hospital plotted against the average of its ten nearest neighbors under Euclidean distance between hospital profiles. The fact that the average strongly correlates with the original quality rating shows that the embeddings of the hospitals remain close if they are close under Euclidean distance.
Figure 9 shows that the back propagation neural net satisfies Condition 2 of from (5). The histogram on the left shows the DNN affinity between hospitals with different initial quality ratings (i.e. when ). To satisfy Condition 2, (i.e. ). Also, for the histogram on the right we define
The histogram on the right shows . The smaller the ratio as approaches , the stronger the influence of the initial ranking on the final affinity.
Moreover, we can compare the contraction guaranteed by (8). For the hospital ratings,
which makes the right hand side of (8) equal .
Figure 10 indicates that the DNN metric from (5
) gives a better notion of small neighborhoods than a simple Euclidean metric. Each metric defines a transition probability
. For each point , the plot finds(12)  
subject to 
It is important for (12) to be small, as that implies the metric generates tightly clustered neighborhoods. As one can see from Figure 10, the DNN metric creates much more tightly clustered neighborhoods than a Euclidean metric. Figure 11 gives summary statistics of these neighborhoods for varying diffusion scales.
Another positive characteristic of our DNN embedding is the reduced local dimension of the data. Figure 12 plots the eigenvalues of the diffusion kernels
, where the Markov chain
describes either comes from the Euclidean metric or the DNN metric. These eigenvalues give us information about the intrinsic dimension of the data, as small eigenvalues do not contribute to the overall diffusion. To compare the eigenvalues across different diffusion kernels, we normalize the eigenvalues by setting, as this is the average time it takes to diffuse across the system. Cutting off the eigenvalues to determine the dimension is fairly arbitrary without a distinct drop off, but an accepted heuristic is to set the cutoff at
With this definition of dimension, dim for the DNN metric embedding, whereas dim for the Euclidean metric embedding.
4.3.2 Examples of “Best” and “Worst” Hospitals
Row sum ranking is a poor metric for differentiation between most hospitals, but it is able to distinguish the extremal hospitals. In other words, hospitals with almost all features considerably above the mean would have obviously been assigned a 10 within any ranking system, and hospitals with almost all features below the mean would have obviously been assigned a 1 by any ranking system.
Let us take the best and worst hospitals under the row sum ranking, and consider how well these correspond to the final DNN rankings from Section 4.2. The average quality function value across the top row sum rankings is 8.03, and the average quality function value across the bottom row sum rankings is 1.75.
While these averages are indicative of the agreement between the DNN ranking and simple understandings of hospital quality, they are not perfect. For example, the top row sum hospitals have an average DNN ranking of 8.03 because one of these hospitals has a DNN ranking of 4.84. This hospital is an interesting profile. Qualitatively, its process features and survey features are above average, but their outcomes are significantly worse than the mean. Even though only 16 of its 81 features are below the mean, the features for mortality from heart failure and pneumonia, and several measures of hospital associated infection, are all more than 1 standard deviation below the mean. As all five of those features are heavily weighted by the algorithm and deemed crucial by experts, the hospital is assigned a low ranking despite having a strong row sum ranking.
Figure 13 shows the embeddings of hospitals without the neural net step. In other words, we simply weight the mortality and readmission features and calculate an affinity based off those feature vectors. There are two important details. First, the embedding is very unstructured, implying a lack of structure within the naive affinity matrix. Second, while the cloud doesn’t necessarily reflect structure, it does give a simple understanding of local neighborhoods.
Figure 13 also shows that our new notion of rankings reflects a naive understanding of hospital quality. Namely, local Euclidean neighborhoods maintain the same ranking. Also, the 10 best and worst hospitals under the row sum ranking are circled. As expected, they exist on the two extremes of the embedding cloud, and our rankings agree with their notion of quality.
4.3.3 Dependence on Initial Rankings
Another important feature of a ranking algorithm is its robustness across multiple runs. To demonstrate stability of the neural net section of the algorithm, we run multiple experiments with random parameters to test the stability of the quality function. Figure 14 shows the quality rating from one run of the algorithm (ie. the average ranking after 100 iterations of the neural net) against the quality rating from a separate run of the algorithm. In both runs, hospitals from are randomly chosen for training the model and then tested on the other hospitals in . Figure 14 shows the rankings for all hospitals across both runs. Given the strong similarity in both quality rating and overall organization, it is clear that the average over 100 iterations of a neural net is sufficient to decide the quality function.
It is also important to examine the dependence our initial binning and ranking has on the final ranking of the hospitals. Clearly the initial binning is only meant to give approximate ranks, so a strong dependence on these rankings would be problematic. Table 1
shows the confusion matrix between the initial hospital rankings and the final rankings assigned from the neural net. The purpose of the neural net second step is to reclassify hospitals that are binned incorrectly due to spurious correlations.
Final  Rank  


8  11  0  0 
First Rank  15  730  129  0 
0  97  330  219  
0  0  27  48 
Confusion matrix between initial ranking on hospital bins and final ranking from neural net. For simplicity, rankings have been rounded into quartiles for purposes of confusion matrix.
Let us examine a few of the hospitals that made the biggest jump. One of the hospitals increased from a rating of 3 to a DNN rating of 6.55. Several of its process features are well below average with a couple 2 standard deviation below average, and its survey features are average to slightly below average. For this reason, it was immediately classified as a poor hospital. However, its readmission features across the board are above average, with readmission due to heart failure 1.5 standard deviations above average. Also, mortality due to heart failure is a standard deviation above average, and all features of hospital associated infection are above the mean as well. In the initial ranking of the pseudohospitals, the fact that there are 29 survey features as compared to 5 readmission features brought the weighting down, even when giving extra weight to the readmission features. After the neural net ranking, the ranking is improved to 6.55.
Another hospital dropped from a rating of 7 to a DNN rating of 3.4. Its process feature are all above average, and its readmission for heart failure is 1.5 standard deviations above the mean. Because readmission is weighted initially and there are a large number of process measures, the hospital was initially binned with better hospitals. However, its features for hospital associated infections, as well as mortality due to pneumonia and heart failure are a standard deviation below the mean, and its average survey score is a standard deviation below the mean as well. For these reasons, the DNN ranking dropped significantly.
4.3.4 Affinity Dependence on Individual Features
As another validation mechanism, we consider the smoothness of each feature against the metric . For each feature , we estimate the Lipschitz constant such that
where is simply the Euclidean distance in the diffusion embedding space of the DNN affinity matrix . Once is normalized by range, gives a measure of the smoothness of under our new metric.
Lipschitz Constant Under
Feature  Lipschitz Constant 

Quality Rating Function  0.124 
READM 30 HOSP WIDE  0.303 
MORT 30 HF  0.346 
READM 30 HF  0.350 
MORT 30 AMI  0.371 
MORT 30 PN  0.384 
READM 30 PN  0.398 
READM 30 AMI  0.430 
READM 30 HIP KNEE  0.440 
H HSP RATING 0 6  0.582 
H RECMND DN  0.595 
H COMP 1 SN  0.618 
H HSP RATING 9 10  0.620 
H COMP 3 SN  0.688 
H RECMND DY  0.691 
H COMP 1 A  0.712 
H QUIET HSP SN  0.735 
VTE 5  0.740 
PSI 90 SAFETY  0.758 
H COMP 2 SN  0.758 
H COMP 4 SN  0.761 
COMP HIP KNEE  0.762 
STK 8  0.763 
ED 1b  0.766 
OP 20  0.789 
ED 2b  0.795 
PSI 4 SURG COMP  0.807 
VTE 3  0.807 
H RECMND PY  0.811 
H HSP RATING 7 8  0.831 
OP 21  0.832 
STK 1  0.834 
HAI 5  0.838 
H CLEAN HSP SN  0.839 
H COMP 5 SN  0.844 
HAI 2  0.846 
HAI 3  0.868 
H COMP 6 Y  0.870 
H COMP 6 N  0.870 
HAI 1  0.875 
STK 6  0.889 
H COMP 4 A  0.898 
PC 01  0.904 
VTE 2  0.906 
Feature  Lipschitz Constant 

H COMP 1 U  0.908 
OP 11  0.913 
OP 9  0.913 
H COMP 3 A  0.918 
H COMP 2 A  0.931 
OP 13  0.934 
OP 18b  0.935 
STK 10  0.942 
H QUIET HSP A  0.946 
OP 10  0.978 
H COMP 5 A  0.996 
H CLEAN HSP A  1.023 
HF 3  1.077 
AMI 2  1.081 
HAI 6  1.088 
SCIP INF 9  1.102 
VTE 4  1.106 
AMI 10  1.133 
OP 6  1.147 
H COMP 2 U  1.186 
SCIP CAR  1.192 
VTE 1  1.221 
H CLEAN HSP U  1.225 
OP 22  1.227 
H QUIET HSP U  1.258 
SPP  1.264 
IMM 1A  1.272 
IMM 2  1.284 
STK 2  1.288 
OP 7  1.291 
H COMP 4 U  1.325 
H COMP 3 U  1.365 
HF 1  1.430 
STK 5  1.443 
PN 3b  1.545 
H COMP 5 U  1.566 
SCIP INF 2  1.994 
SCIP INF 3  2.001 
PN 6  2.066 
SCIP INF 1  2.131 
SCIP VTE  2.151 
SCIP INF 10  3.333 
HF 2  5.111 
From the Lipschitz constant, we can conclude that the most influential features are the mortality and readmission scores, followed closely by survey scores and post surgical infection scores. While the process features have some smoothness, the Lipschitz constants are significantly larger than outcome and some survey scores.
Almost all the hospitals in Figure 15 have an average process score within a half standard deviation of the mean, which is well within acceptable levels. For this reason, these features are not as strong of features as some of the other features. However, the few hospitals that are well below the mean have middle to poor quality ratings.
Figures 16 and 17 show the embedding of the hospitals colored by their average survey and average outcome features. These are more strongly correlated with the overall quality function.
Note that the average across all features from a category is not the ideal description of that category. For a given hospital, there can be significant variability within a category which would not be captured by the mean. The category average is only meant to demonstrate a general trend. Figure 18 shows a weighted average of the readmission and mortality features, with the weights determined by a least squares fit with the DNN quality function as a dependent variable.
We can also use these embeddings, along with Figure 19, to show why the embedding has the two prongs on the left side of the image among the well ranked hospitals. Notice from Figure 19 that readmission is very strong along the lower prong, while mortality scores are very strong along the upper prong. Moreover, there is some level of disjointness between the two scores [10].
This begs the question of why the strong readmission cluster is ranked higher than the strong mortality cluster. That can be explained in Figure 16. The survey features are much stronger along the lower cluster with high readmission than they are in the mortality cluster. This explains the cluster of hospitals receiving top quality ratings despite having average to slightly above average mortality scores. The same can be said, to a lesser degree, for the process scores in Figure 15. So the differentiation in rankings is derived from the fact that, at their highest levels, more of the features agree with readmission features than they do with mortality features.
The left plot in Figure 20 represents each hospital with three features: the mean score across process features, the mean score across survey features, and the mean score across outcome features. The plot is colored by the quality function, and it is clear that the rankings reflect the trend of these three average features. The right plot in Figure 20 represents each hospital with the NNLS process, survey, and outcome scores. Table 2 list those features that have nontrivial weights with pvalue .
Features  Coefficient Magnitude 
HAI 3  0.029 
HAI 6  0.010 
READM 30 AMI  0.148 
READM 30 HF  0.139 
READM 30 HIP KNEE  0.148 
READM 30 HOSP WIDE  0.122 
READM 30 PN  0.074 
PSI 4 SURG COMP  0.018 
PSI 90 SAFETY  0.003 
MORT 30 HF  0.116 
MORT 30 PN  0.121 
MORT 30 AMI  0.072 
SPP  0.053 
OP 10  0.045 
OP 11  0.080 
OP 13  0.003 
AMI 10  0.069 
HF 1  0.058 
HF 3  0.136 
STK 1  0.136 
STK 5  0.040 
STK 6  0.156 
STK 8  0.059 
STK 10  0.083 
VTE 2  0.183 
VTE 3  0.126 
VTE 4  0.002 
PN 6  0.112 
SCIP CAR  0.074 
IMM 2  0.239 
OP 6  0.077 
OP 7  0.009 
OP 21  0.028 
PC 01  0.108 
H CLEAN HSP SN  0.171 
H COMP 1 SN  0.141 
H COMP 2 SN  0.082 
H COMP 5 SN  0.119 
H COMP 6 Y  0.156 
H HSP RATING 7 8  0.185 
H RECMND DN  0.176 
H RECMND PY  0.123 

5 Conclusion
We introduced an algorithm for generating new metrics and diffusion embeddings based off of expert ranking. Our algorithm incorporates both data point geometry via hierarchical diffusion geometry and nonlinear metafeatures via stacked neural nets. The resulting embedding and rankings represent a propagation of the expert rankings to all data points, and the resulting metric generated by the stacked neural net gives a Lipschitz representation with respect to Euclidean distance that learns important and irrelevant features according to the expert opinions and in automated fashion.
Although the ranking algorithm seems tied to the process of expert rankings of hospitals, the underlying idea of generating metrics in the form of (5). and propagating first pass rankings to the rest of the data, is quite general. For example, this method could be used to propagate sparsely labeled data to the rest of the dataset, with expert bin rankings replaced by the mode of the labelled data in the bin.
This method also touches on the importance of incorporating data point organization into the neural net framework when dealing with smaller data sets and noisy features. Without the expert driven function to roughly organize the data points, the stacked autoencoder fails to determine any relevant features for separation, as shown in Figure 7.
We will examine the implications of our hospital ratings on health policy, as well as discuss the various types of hospitals in our embedding, in a future paper. Future work will also examine further examination of the influence of data point organization on neural nets and the generation of metafeatures. Also, it would be interesting to examine other applications of propagation of qualitative rankings and measures.
Acknowledgments
The authors would like Arjun K. Venkatesh MD, MBA, MHS and Elizabeth E. Drye MD, SM for helping to develop the initial rankings of the pseudohospitals, Uri Shaham for use of his deep learning code, and Ali Haddad for providing the base code for the questionnaire. Alexander Cloninger is supported by NSF Award No. DMS1402254.
References
 [1] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. IEEE Transactions on Neural Computation, 2003.
 [2] Yoshua Bengio. Deep learning of representations: Looking forward. Statistical Language and Speech Processing, 2013.
 [3] E.H. Bradley, J. Herrin, B. Elbel, and et al. Hospital quality for acute myocardial infarction: Correlation among process measures and relationship with shortterm mortality. JAMA, 2006.
 [4] Y. Cho and LK Saul. Kernel methods for deep learning. Advances in neural information processing systems, 2009.
 [5] Ronald R. Coifman and Matan Gavish. Harmonic analysis of digital data bases. Wavelets and Multiscale analysis, 2011.
 [6] Ronald R Coifman and Matthew J Hirn. Diffusion maps for changing data. Applied and Computational Harmonic Analysis, 2013.
 [7] Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21(1):5–30, 2006.
 [8] Center for Medicare and Medicaid Services (CMS). Hospital compare. CMS website: www.medicare.gov/hospitalcompare/, Access Date: June 15, 2015.
 [9] Ali Haddad, Dan Kushnir, and Ronald R. Coifman. Texture separation via a reference set. Applied and Computational Harmonic Analysis, 2014.
 [10] HM Krumholz, Z Lin, PS Keenan, and et al. Relationship between hospital readmission and mortality rates for patients hospitalized with acute myocardial infarction, heart failure, or pneumonia. JAMA, 2013.
 [11] Prasanta Chandra Mahalanobis. On the generalized distance in statistics. Proceedings of the National Institute of Sciences, 1936.

[12]
Gregorie Montavon, Mikio Braun, and KlausRobert Muller.
Kernel analysis of deep networks.
Journal of Machine Learning Research
, 2011.  [13] R. Rojas. Neural networks: a systematic introduction. Springer Science and Business Media, 1996.
 [14] Will E. Leeb Ronald R. Coifman. Earth mover’s distance and equivalent metrics for spaces with hierarchical partition trees. Yale CS Techinical Report, 2013.

[15]
L. Saul and S. Roweis.
Think globally, fit locally: unsupervised learning of nonlinear manifolds.
Journal of Machine Learning Research, 4(12):119–155, 2003. 
[16]
Amit Singer and Ronald R Coifman.
Nonlinear independent component analysis with diffusion maps.
Applied and Computational Harmonic Analysis, 25(2):226–239, 2008.
Comments
There are no comments yet.