The vast amounts of data collected by healthcare providers in conjunction with modern data analytics techniques present a unique opportunity to improve the quality and safety of medical care, for patient benefit. In the United Kingdom, the National Health Service (NHS) has a long history of documenting extensively the different aspects of healthcare provision. The NHS is currently in the process of increasing the availability of several databases, properly anonymised, with the aim to leverage advanced analytics to identify areas of improvement in its services. One such resource is the National Reporting and Learning System (NRLS), a central repository of patient safety incident reports from England and Wales collected since 2004, which now contains more than 13 million detailed records. The incidents are reported using a set of standardised categories and contain a wealth of organisational and spatiotemporal information (structured data) as well as, crucially, a substantial component of free text (unstructured data). The incidents are wide ranging: from patient accidents to lost forms or referrals; from delays in admission and discharge to serious untoward incidents, such as retained foreign objects after operations. The review of such data provides critical insight into complex procedures in healthcare with an aim towards service improvement.
Although statistical analyses are routinely performed on the structured components of the data, the free text component remains largely unused. Free text can be read manually but this task is time consuming, hence often ignored in practice. Methods that provide automatic, content-based categorisation of incidents from the free text could help sidestep difficulties in assigning incident categories from a priori pre-defined lists, reducing human error and burden, as well as offering a unique insight into the root cause analysis of incidents that could improve safety, quality of care and efficiency.
Here, we showcase an algorithmic methodology that detects content-based groups of records in an unsupervised manner, based only on the free, unstructured textual description of the incidents. To do so, we combine deep neural-network high-dimensional text-embedding algorithms (Doc2vec) with network-theoretical methods (multiscale Markov Stability (MS) community detection) applied to a sparsified geometric similarity graph of documents derived from text vector similarities.
Traditional natural language processing tools have generally used bag-of-words representations of documents followed by statistical methods based on Latent Dirichlet Allocation (LDA) to cluster documents. More recent approaches have used deep neural network based language models, without a full multiscale graph analysis(Hashimoto et al., 2016), while previous applications of network theory to text analysis (Lancichinetti et al., [n. d.]) were carried out at a single scale and used bag-of-word arrays lacking the power of neural network text embeddings. In contrast, multiscale community detection allows us to find groups of records with consistent content at different levels of resolution; hence the content categories emerge from the textual data, rather than fitting to pre-designed classifications.
Our analysis starts by training a Doc2vec vector text embedding using the 13 million NRLS records. (We have also trained models to 1 and 2 million records and the results are similar.) Once the text embedding is obtained, we use it to produce document vectors for a subset of 3229 incident reports from St Mary’s Hospital, London (Imperial College Healthcare NHS Trust) over three months in 2014. Our graph clustering method is then applied to cluster these records across different levels of resolution, revealing multiple levels of intrinsic structure in the topics of the dataset, as shown by the extraction of relevant word descriptors from the groups of records.
Upon reporting by the operator, the records had been independently hand-coded with up to 170 features per record, including a two-level manual classification of the incidents: 15 categories at Level 1; 95 sub-categories at Level 2. We carried out an a posteriori comparison against the hand-coded categories assigned by the reporter. Several of our content-based clusters exhibit good correspondence with well-defined hand-coded categories at both levels; yet our results also provide further resolution in certain areas and a complementary characterisation of the incidents, not defined in the a priori external classification. We find that our methodology provides improved performance over LDA models, as quantified by the Uncertainty Coefficient against the hand-coded categories.
Text Pre-processing and Doc2Vec Model Training. The pre-processing of the raw text consisted of: lowering capital letters; tokenising sentences into words; stemming; and removing punctuation, stop-words and all numeric tokens. We then trained Doc2Vec models (Le and Mikolov, 2014) using the PV-DBOW method from the Gensim (Rehurek and Sojka, 2010) library. To ascertain the effect of the training of the Doc2Vec model on performance, we repeated the training with a broad range of parameters sets (window size, minimum count, subsampling). We carried Doc2Vec training both on a standard (generic, non-specialised) Wikipedia English corpus and on the full NRLS dataset (13+ million records with specialised language). Table 1 shows that, while the Wikipedia corpus is useful to train models when the parameters sets are weak, the NRLS dataset performs better for optimised parameter sets. Once optimised, the Doc2Vec model trained on the NRLS corpus is used to infer vectors for each of the records in our analysis dataset.
|Model Parameters||Training Corpus|
Graph Construction. We constructed a normalised similarity matrix
based on the cosine similarity by: computing the matrix of cosine similarities between all pairs of records,; transforming it into a distance matrix ; applying element-wise max norm to obtain ; and obtaining the normalized similarity matrix which has values in . This (full) similarity matrix can be viewed as a completely connected, weighted graph. However, such a graph contains many edges with small weights (i.e., weak similarities) since, in high dimensional noisy datasets, even the least similar nodes present a non negligible degree of similarity. We thus apply a simple geometric sparsification to the normalized distance matrix
using the MST-kNN method(Veenstra et al., 2017)
, a geometric heuristic that preserves the global connectivity of the graph while retaining the local geometry of the dataset. The MST-kNN graph is a weighted graph obtained by the union of the minimum spanning tree (MST) of, and adding edges connecting each node to its nearest nodes (kNN). We scanned for the graph construction and found that the MST-kNN graph with presents a reasonable balance between local and global structure in the dataset, and thus analyse it with the multi-scale graph partitioning framework. However, the results are robust to the choice of within a set of values. Note that the MST-kNN method avoids global similarity thresholding.
Markov Stability Multiscale Graph Partitioning. We apply Markov Stability (MS), a multiscale community detection method, to the MST-kNN graph in order to detect clusters of documents with similar content at different levels of resolution. MS is an unsupervised method that scans across all scales to detect robust and stable graph partitions using a continuous time diffusion process on the graph. The method does not need to choose a priori the number or type of relevant subgraphs, other than it retains the diffusive flow over the Markov time . Hence acts as a resolution parameter revealing relevant partitions that persist over particular time scales in an unsupervised manner. For more details see (Delvenne et al., 2010; Schaub et al., 2012; Lambiotte et al., 2014; Bacik et al., 2016).
Briefly, the method optimises the MS function over the space of graph partitions at each time . MS is defined as the trace of the clustered autocovariance matrix of the diffusion process (1):
where is the membership matrix of the partition, is the random walk Laplacian of the graph, is the steady-state distribution of the process and . Our method searches for the partition that maximises at each Markov time. The partition is formed by communities (subgraphs) that tend to preserve the flow within themselves over time , since in that case the diagonal (off-diagonal) elements of will be large (small). Although the maximization of (1) is NP-Hard, there are optimisation methods that work well in practice. Here we use the Louvain Algorithm (Blondel et al., 2008) which is efficient and known to give good results for benchmarks. We look to obtain robust partitions, i.e., partitions that are relevant across scales (i.e., consistently found over extended Markov time) and highly reproducible (i.e., consistently found by the Louvain optimisation). This is achieved by running the Louvain algorithm 500 times with different initialisations at each Markov time, picking the 50 with the highest MS value, and computing the variation of information (Meilă, 2007) of this ensemble. In addition, we also compute the variation of information between the optimised partitions found across time, . Robust partitions are indicated by dips of and extended plateaux of , indicating a partition that is robust to the optimisation and valid over extended scales (Bacik et al., 2016; Lambiotte et al., 2014).
Visualization and interpretation of results.
Tracking membership through Sankey diagrams. Sankey diagrams visualise the flow of node memberships across different partitions and categories. We use Sankey diagrams with two different objectives: (i) a multilayer Sankey diagram to represent the results of the multi resolution MS community detection across different scales (Fig. 1); (ii) two-layer Sankey diagrams to indicate the correspondence between MS clusters and the hand-coded external categories at a given level of resolution (Fig. 2).
Normalized contingency tables.2). We score the quality of this correspondence using the uncertainty coefficient, an informational theoretical measure of similarity between groupings.
Word clouds of increased intelligibility through lemmatisation. To visualise the content of the document clusters, which can be understood as a type of topic detection, we used Word Clouds weighted through lemmatisation as an intuitive way to summarise content and compare a posteriori with hand-coded categories. Word clouds can also provide an aid for monitoring when used by practitioners.
We applied full MS across an extended, finely sampled span of Markov times (0.01–100 in steps of 0.01) to the similarity MST-kNN graph of 3229 NRLS incident records. Figure 1 presents a summary of this analysis including the number of clusters and the two metrics of variation of information across all Markov times. The existence of several long plateaux in coupled to the existence of dips in the imply the presence of robust partitions at different levels of resolution. We choose several robust partitions, from finer to coarser, and examine their structure and content. Their relative node membership is shown with a multi-level Sankey diagram.
Quasi-hierarchical structure: themes at different resolution levels. The MS analysis in Figure 1 reveals a rich multi-level structure of partitions, with a strong quasi-hierarchical organization. It is important to remark that, although the Markov time acts as a natural resolution parameter from finer to coarser partitions, our process of optimization does not impose such a hierarchical structure. Hence this organisation is intrinsic to the data and implies the existence of content communities which naturally integrate with each other as sub-themes of larger thematic categories. The detection of intrinsic scales within the graph at which robust partitions exist, thus allows us to obtain thematically-based clusters of records at different levels of resolution.
Interpretation of the MS communities: Word clouds and a posteriori comparison against hand-coded categories. To ascertain the relevance and relationship between the layers of MS clusters, we examined in detail the five levels of resolution in Figure 1. For each level, we prepared word clouds (lemmatised for increased intelligibility) and a Sankey diagram and contingency table linking content clusters with the hand-coded categories assigned by the operator. Note that this comparison was only done a posteriori, i.e., the external categories were not used in our analysis, hence our approach is truly unsupervised. As an example, Figure 2 shows the 17-community partition with word clouds for all clusters and the comparison with the 15 hand-coded categories in Level 1.
Comparison of the MS content clusters against other NLP methods. We compared the MS document partitioning results against LDA models with a range of similar number of topics. LDA models are trained and inferred using the Gensim module on the same set of documents. We then quantify the match of the MS clusters with the hand-coded categories using the Uncertainty Coefficient (White et al., 2004)
where is the entropy of the hand-coded categories and
is the entropy of the clustering. The MS clusters show improved correspondence with both Level 1 and 2 categories as compared to LDA and spectral clustering methods (Fig.3).
This work has applied a multiscale graph partitioning algorithm (MS) to determine topic clusters for a textual dataset of healthcare safety incident reports in an unsupervised manner at different levels of resolution. The method uses paragraph vectors to represent the records and obtains an ensuing similarity graph of documents constructed from their content. This method brings the advantage of multi-resolution algorithms capable of capturing clusters without imposing a priori the number or structure of the clusters. Furthermore, it selects different levels of resolution of the clustering to suit the requirements of each task depending on the level of detail. The a posteriori analysis against hand-categories showed that the method recovers meaningful categories and outperformed LDA at both categorisation levels. Furthermore, some of the MS content clusters capture topics of medical relevance, which provide complementary information to the external classifications. The nuanced information and classifications extracted from free text analysis suggest a complementary axis to existing approaches to characterise patient safety incident reports, as the method allows for the discovery of emerging topics or classes of incidents directly from the data when such events do not fit the pre-assigned categories.
- Bacik et al. (2016) K A Bacik, M T Schaub, M Beguerisse-Díaz, Y N Billeh, and M Barahona. 2016. Flow-Based Network Analysis of the Caenorhabditis elegans Connectome. PLOS Computational Biology 12, 8 (2016), 1–27.
- Blondel et al. (2008) V D Blondel, J-L Guillaume, R Lambiotte, and E Lefebvre. 2008. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment 2008, 10 (2008), P10008.
- Delvenne et al. (2010) J-C Delvenne, S N Yaliraki, and M Barahona. 2010. Stability of graph communities across time scales. Proceedings of the National Academy of Sciences of the United States of America 107, 29 (7 2010), 12755–60.
Hashimoto et al. (2016)
K Hashimoto, G
Kontonatsios, M Miwa, and S
Topic detection using paragraph vectors to support active learning in systematic reviews.Journal of Biomedical Informatics 62 (8 2016), 59–65.
- Lambiotte et al. (2014) R Lambiotte, J C Delvenne, and M Barahona. 2014. Random Walks, Markov Processes and the Multiscale Modular Organization of Complex Networks. IEEE Transactions on Network Science and Engineering 1, 2 (7 2014), 76–90. https://doi.org/10.1109/TNSE.2015.2391998
- Lancichinetti et al. ([n. d.]) A Lancichinetti, M I Sirer, J X Wang, D Acuna, K Körding, and L N Amaral. [n. d.]. High-Reproducibility and High-Accuracy Method for Automated Topic Classification. Phys. Rev. X 1 (jan [n. d.]), 11007. https://doi.org/10.1103/PhysRevX.5.011007
- Le and Mikolov (2014) Q Qv Le and T Mikolov. 2014. Distributed Representations of Sentences and Documents. International Conference on Machine Learning - ICML 2014 32 (2014), 1188–1196. https://doi.org/10.1145/2740908.2742760
- Lupton and Leonardo (2017) R Lupton and Leonardo. 2017. ricklupton/sankeyview: v1.1.7. https://doi.org/10.5281/zenodo.1098904
Comparing clusterings—an information based
Journal of Multivariate Analysis98, 5 (5 2007), 873–895.
- Rehurek and Sojka (2010) R Rehurek and P Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45–50.
- Schaub et al. (2012) M T Schaub, J C Delvenne, S N. Yaliraki, and M Barahona. 2012. Markov dynamics as a zooming lens for multiscale community detection: Non clique-like communities and the field-of-view limit. PLoS ONE (2012). https://doi.org/10.1371/journal.pone.0032210
- Veenstra et al. (2017) P Veenstra, C Cooper, and S Phelps. 2017. Spectral clustering using the kNN-MST similarity graph. In 2016 8th Computer Science and Electronic Engineering Conference, CEEC 2016 - Conference Proceedings. Institute of Electrical and Electronics Engineers Inc., 222–227. https://doi.org/10.1109/CEEC.2016.7835917
- White et al. (2004) JV White, Sam Steingold, and CG Fournelle. 2004. Performance metrics for group-detection algorithms. Proceedings of Interface 2004 (2004).