Evidence synthesis in the form of Clinical Practice Guidelines (CPGs) serves as a basis for evidence-based decision making in clinical practice. To leverage the knowledge in CPGs for clinical decision support systems, e.g. for integration with electronic health records or automated evaluation of adherence to guidelines, machine-readable versions of CPGs are necessary. However, CPGs today are disseminated mostly as free-text documents, with few formal elements. Thus, Natural Language Processing (NLP) might be helpful to automatically extract information from the unstructured texts and transform them into a structured, or even executable, format. As CPGs are also specific to their country of origin, they are usually published in the respective native language, so NLP technology has to be adapted properly.
A major source for the progress in NLP research in the recent years is the public availability of large text corpora. For documents originating from a clinical context, the protection of personal information is a major requirement for accessibility to researchers. Some corpus initiatives, e.g., i2b2 , Mimic III , or Clef eHealth  make de-identified clinical document collections available under the conditions of Data Use Agreements (DUA). Besides, databases of biomedical research articles like PubMed provide an abundant amount of examples for medical language. However, with only few exceptions, such open-access text corpora are hardly available for the German  and other non-English languages. As of today, there is no viable solution for sharing even de-identified clinical texts in Germany.
In order to address (1) the lack of available German medical text resources for NLP research, and (2) the need for machine-readable CPGs, we constructed a corpus based on a set of German CPGs for oncology. The German Guideline Program in Oncology (GGPO) , operated by the Association of the Scientific Medical Societies in Germany, the German Cancer Society and the German Cancer Aid, is in a unique position to enable this research, as their guidelines are also provided via a mobile app . Hence, the data set is available in a semi-structured format with rich, formatted metadata, resulting in a much higher data quality than data extracted a posteriori from PDF versions of the guidelines.
The GGPO guidelines are available free of charge and do not contain sensitive data about individual patients. However, as processing of the guidelines is by default still prohibited by the issuer’s copyright, we provide access to the pre-processed data for other researchers via a DUA.
|Corpus / Data||Documents||Sentences||Tokens||Available|
|FraMed: clinical reports and medical textbook snippets ||–||6k||100k||✗|
|Reports from five medical domains ||544||–||–||✗|
|Radiology reports ||174||4k||28k||✗|
|Transthoracic echocardiography reports ||140||–||–||✗|
|Operative reports (surgery) ||450||22k||266k||✗|
|Discharge summaries from a dermatology department ||1,696||–||–||✗|
|Discharge summaries and clinical notes from nephrology domain ||1,725||28k||158k||✗|
|Discharge summaries and clinical notes from nephrology domain ||183||2k||13k||✗|
|X-ray reports ||3,000||–||–||✗|
|3000PA: internistic and ICU discharge summaries ||3,000||–||–||✗|
|3000PA Jena Part||1,006||170k||1,421k||✗|
|JSynCC: case examples from medical textbooks  (v1.1)||903||29k||368k||✓|
|Discharge summaries with osteoporosis diagnosis ||1,982||–||2,001k||✗|
|GGPOnc – recommendations||25 (4,348)||7k||132k||✓|
|GGPOnc – complete corpus||25 (8,418)||60k||1,340k||✓|
Ii Related work
Due to legal data protection regulations, the availability of German-language clinical text corpora is severely restricted — most clinical corpora are only accessible to the research staff within the lifetime of a project and remain inaccessible forever for the outside world. There have been a few disconnected activities in the German NLP community to create in-project clinical corpora. In Table I we list, to the best of our knowledge, all existing German-language clinical research text corpora with clinical documents or collections of case reports that have been described in scientific publications. In addition to pure clinical documents, other document types are also interesting for the NLP community, e.g. CPGs.
CPGs as a target for automated text analytics have been much less utilized compared to other scientific publications and clinical documents. Most of that work took place in the context of formalizing CPGs as computer-interpretable guidelines . Bouffier and Poibeau  describe an approach to fill in a semi-structured Guideline Elements Model template by segmenting unstructured guidelines using linguistic patterns. An evaluation was run on 18 French guidelines. Serban et al.  describe the extraction and instantiation of linguistic templates for guideline formalization, evaluated on a Dutch guideline for breast cancer treatment. German CPGs were the focus of Becker and Bockmann  who adapted Apache cTakes to detect German UMLS concepts and evaluated their approach on a single German breast cancer guideline. Zadrozny et al.  outline a system which identifies contradictions and disagreements in English CPGs.
Some authors have focused on extracting more task-specific information, such as activities , process structures [44, 48, 17] or negation triggers. Most of these approaches work with relatively small annotated corpora and English language, only. Recently, Fazlic et al.  use LSTMs and fuzzy rules to extract “action takers”, “symptoms”, “actions” and “purposes” from CPGs, recognize recommendations and predict the grade of recommendation. The authors use a data set extracted from PDF versions of 45 guidelines with 1,020 recommendations. Some larger corpora of CPGs in the English language exist already: Hussain et al.  present the Yale Guideline Recommendation Corpus (YGRC), a sample of 1,275 guideline recommendations extracted from National Guideline Clearinghouse (NGC). Their work revealed inconsistencies in writing style and reporting of the strength of recommendations. Using a subset of YGRC, Gad El-Rab et al.  present a rule-based approach to detect procedures and drug recommendations. Read et al.  describe the CREST corpus, consisting of 4,029 recommendations from 170 guidelines annotated with their respective recommendation strength and report a total number of 8,138 types within the recommendations. Large corpora of CPGs lend themselves to mining the state-of-the-art knowledge in a medical subfield. For instance, Leung et al.  identify comorbidities by analyzing pairs of co-occuring conditions, using a corpus of 268 NGC guideline summaries. Leung and Dumontier 
identify drug-disease relations via named entity recognition using a corpus of 377 NGC guideline summaries. The extracted relations are compared to structured drug product labels to assess their overlap.
Iii-a Data Collection
In order to assemble the corpus of German CPGs, we acquired semi-structured JSON versions of the guidelines from the REST API of the Content Management System (CMS) that serves the backend for the mobile app provided by the GGPO. The data was subsequently transformed from JSON to an XML format. We preserved the document structure (chapters and sections), as well as recommendation metadata and literature references. An example of the resulting XML format can be found in Listing 1. The metadata elements are described in Table II.
The guidelines distinguish between recommendations and background texts, and we preserved this distinction in the corpus. In general, the recommendations tend to be concise statements related to a particular clinical question. For evidence-based recommendations, literature references and evidence levels are included. The background texts provide the reasoning behind the recommendations and a summary of the evidence underlying the recommendations, again backed by literature references.
|Recommendation creation date||Date the recommendation was first introduced|
|Type of recommendation||Evidence-based or consensus-based statement or recommendation|
|Recommendation grade||A (strong recommendation)|
|0 (weak recommendation / option)|
|Strength of consensus||Strong Consensus|
|Approved by majority|
|Total vote in percentage||Percentage of approval among the expert committee|
|Literature references||List of evidence backing up the recommendation|
|Expert opinion||Yes or absent|
|Level of evidence||According to Oxford , SIGN, or GRADE|
|Edit state||State (checked, new or modified) and text note regarding guideline updates|
Iii-B Automated Annotation
Besides the XML version of the corpus, we created plain text versions of all recommendation of background text parts to facilitate processing by existing NLP pipelines. For preprocessing, like sentence splitting and tokenization, we used the JCoRe  (i.e., Uima-based) pipelines and FraMed  models, which were developed for German clinical text.
We used the JuFit tool, a filter for UMLS, to create a dictionary of all German words from the UMLS (version 2019AB)111https://www.nlm.nih.gov/research/umls/ and the semantic groups ANAT (Anatomical Structure), CHEM (Chemicals & Drugs), DEVI (Devices), DISO (Disorders), LIVB (Living Beings), PHYS (Physiology), and PROC (Procedures) (without advanced JuFit rules), as well as a list of gene names compiled from Entrez Gene and UniProt with the approach originating from Wermter et al. and German stop words. With these dictionaries, we configured a JCoRe pipeline for a dictionary-based text search.
Finally, we detect TNM expressions222The UICC TNM system is a classification scheme for malignant tumors, see https://www.uicc.org/resources/tnm, which are extracted using a rule-based approach implemented with the Python library spaCy. This part was originally developed for German pathology reports in the context of the HiGHmed consortium of the Medical Informatics Initiative of Germany. TNM expressions and genes were specifically chosen for their relevance in cancer treatment.
|9||Prevention of skin cancer||288||119||2,354||55,965||9,140||952|
|10||Actinic keratosis and SCC of the skin||199||74||2,590||54,073||6,861||1,278|
|14||Prevention of cervix cancer||302||103||2,055||48,676||7,989||1,391|
|15||Renal cell cancer||276||122||2,118||48,013||8,202||1,496|
|19||Chronic lymphocytic leukemia (CLL)||290||138||1,410||34,470||5,682||725|
|21||Hepatocellular cancer (HCC)||157||88||1,296||27,852||5,704||803|
|22||Malignant ovarian tumors||193||94||1,136||25,807||5,110||1,013|
|25||Oral cavity cancer||111||76||630||15,438||3,376||1,026|
Iv-a Corpus Characteristics
In total, 25 GPGs with 8,414 text segments were extracted from the CMS comprising the first version of the corpus (summarized in Table III). We report the total number of recommendations and background text segments, since they serve as the units of analysis for our automated annotation pipelines. While the number of recommendations in GGPOnc is comparable to the CREST corpus, the amount of structured metadata and background text in our corpus is much larger.
Of the approximatly 38k literature references in the corpus, around 20k are unique with roughly 9k explicit links to PubMed. We provide bibliographic details on these references alongside the corpus to facilitate research on the relationships between CPG and the underlying medical evidence. Table IV summarizes the automated entity extraction results. The result quality and their interpretation in comparison to other German (clinical and non-clinical) text corpora will be discussed in the next section.
The whole corpus consists of:
a single XML file including the document structure and all mentioned metadata
a file for the complete literature index
individual plain text versions of the text segments, sentences and tokens
automatically created entity annotations and a subset of manually corrected annotations
As CPG are subject to a regular update cycle, we are able to automatically repeat the data acquisition process in the future to provide a historical view on the guideline development. For instructions on how to get access to the corpus see: https://www.leitlinienprogramm-onkologie.de/projekte/ggponc-english/.
|GGPOnc||Clinical corpora||Non-Clincal corpora|
|Tokens / Sentence||22.5||19.0||8.8||12.5||16.5||20.9||25.3|
Iv-B Comparison with Other German Medical and Non-Medical Corpora
We analyze the characteristics of GGPOnc by comparing the entity matches with three German medical text corpora, namely version 1.1 of the JSynCC corpus (case examples from clinical text books) , the Jena Part of the 3000PA corpus (1006 German discharge summaries) 333Based on the approval by the local ethics committee (4639-12/15) and the data protection officer of Jena University Hospital discharge summaries were extracted from the HIS of the Jena University Hospital and further transformed. as well as abstracts from German case reports from PubMed. In addition, we compare the results to out-of-domain corpora consisting of German Wikipedia articles of wars (WikiWarsDE)  and news articles from the Krauts corpus . The results are summarized in Table IV.
The fraction of stop words is comparable across all medical text corpora, as is the fraction tokens that map to UMLS concepts. As expected, the guideline recommendations contain more medical terms per token than the background text. Compared to the clinical corpora, the guiofficedelines have more instances of the class Living Beings, as they often describe treatment recommendations for certain populations. Notably, the average sentence length is much greater in the clinical guidelines, and in particular in the background text, pointing at the more scientific style of writing prevalent in the guidelines as compared to clinical narratives. TNM expressions occur much more frequently in GGPOnc, which can be attributed to its focus on the oncology domain. Both out-of-domain corpora contain only small amounts of UMLS concepts (apart from the semantic class Living Beings), which indicates a high precision of our entity tagging approach. In Figure 1 we visualize the overlap of unique medical concepts from UMLS found in each of the corpora. While there is a significant overlap between GGPOnc and the clinical corpora, a major fraction of concepts is unique to each corpus. These results suggest that our corpus combined with other clinical text corpora can provide a more comprehensive view on the use of medical language in general than each of the corpora alone.
Iv-C Evaluation of Annotation Results
The automatic annotations for a subset of the CPGs have been independently reviewed by human experts (1 medical doctor and 3 students of medicine, all of them passed their first medical exam) using the Brat annotation tool . Due to restricted resources for manual annotation work, we decided to evaluate on a subset of four (full) guidelines of a small to intermediate size (16k up to 34k tokens). The CPGs were chosen such that they cover a diverse range of topics and percentages of token matches, with a rather high rate of around 6–8% results per token for HCC and pancreatic cancer as opposed to a lower rate of roughly 5–6% for CLL and psycho-oncology. We calculate the inter-annotator-agreement (IAA) using the pair-wise average -score  of instances and tokens. An instance is a single composite annotation unit that consists of one or more tokens, e.g., eingeschränkte Nierenfunktion (limited renal function) denotes an instance with two (German) tokens, eingeschränkte and Nierenfunktion. The agreement subset consists of the five text segments with the largest amount of automatic annotations for each of the four guidelines, resulting in 20 agreement documents with a size of approx. 17k tokens annotated by all annotators. We excluded the gene category from the IAA analysis, due to an apparently large number of false positive pre-annotations. The IAA achieved an average
-score of 0.757 on instances and 0.745 on tokens. Furthermore, we calculated micro-averaged precision and recall values for the automated annotation results, using the complete set of manually reviewed annotations as the gold standard. The results are summarized in TableV. In another annotation study of diagnoses, symptoms and findings on the Jena part of the 3000PA corpus, average -score values of around 0.7–0.8 have shown to be normal for typical clinical entities, e.g., anatomy or disorders in comparison to diagnoses (approx. 0.7), also for pre-annotations. The low IAA value of Physiology is similar to the IAA of 0.5 on the symptoms category of the named study . The UMLS category Living Beings contains a lot of information similar to personal health information. The average IAA value of around 0.9 is similar to average values of an annotation study for the anonymization of German discharge summaries (-score 0.95) .
|IAA Instances||IAA Token||Precision||Recall|
|UMLS Living Being||(LIVB)||.895||.016||.892||.026||.985||.659|
|TNM (rule based)||.823||.099||.742||.149||.806||.915|
|Overall w/o Genes||.757||.027||.745||.027||.928||.420|
Pair-wise average F1-score and standard deviation () for instance and token based inter-annotator-agreement (IAA), precision and recall per entity class. Genes had to be excluded from IAA analysis due to the overwhelming number of false positives.
V Discussion & Limitations
While the initial results of the information extraction pipelines we employed are promising, there is much room for improvement. The extraction of genes suffers from a large number of false positives, as there are many common German words (e.g., gilt, dar) and three-letter-acronyms (e.g., CLL, HCC) with strings identical with gene names in our large dictionary (around 562k entries). Thus, augmenting the dictionary-based approach with well-known improvements employed in gene taggers for English texts is one of the next steps.
The German UMLS has a number of issues, which severely affect our dictionary-based entity extraction pipelines. First and foremost, its vocabulary size is very limited. For instance, the English UMLS contains over 6.5M entries and the Spanish one around 750k, whereas there are only around 234k entries in the German version (3.6% of the English version). Recently introduced drugs are missing in the UMLS Chemistry category, so a more up-to-date dictionary of drug names should be used for future work. Moreover, use of German umlauts is inconsistent in UMLS, e.g., ä is sometimes transcribed as ae, as in eingeschraenkte Nierenfunktion, which results in a higher than necessary false negative rate. All of these factors contribute to rather low recall values, as evident in Table V.
The accuracy of our dictionary matches is affected by inconsistencies in the use of compounds across the corpus. For instance, Pankreaskarzinompatienten (patients with pancreas carcinoma) would not be detected as an entity, whereas Pankreaskarzinom-Patienten would be, as two entities (Disorders and Living Beings), respectively. In this case, we would choose to annotate the whole compound as Living Beings to avoid annotation on a subword level, which could be addressed using a more finely adapted tokenization algorithm. While precision and recall of our rule-based TNM extraction approach are high on GGPOnc, one has to be careful as certain TNM expressions can have a completely different meaning in another context (ambiguity). For instance, V1 and V2 are valid TNM components referring to venous invasion, but are also detected in the WikiWarsDE corpus when actually referring to German missiles from World War II.
We presented GGPOnc, one of the largest corpora of German medical text to date, assembled from the German CPGs in oncology and equipped with rich structure and metadata. We applied clinical information extraction pipelines to extract a variety of entity classes. Despite the limitations we discussed, the information extracted so far can be of immediate use to enable semantic search functionalities in the guideline app  or in clinical decision support systems . Our results indicate that GGPOnc
shares many characteristics with existing clinical text corpora. This can facilitate the development of machine learning-based NLP algorithms for German clinical text. Beam et al. suggest that combining corpora covering different parts of medical terminology can improve the utility of trained word embeddings. In addition to the German documents discussed in this work, some of the GGPO guidelines have an additional English version, which could be used to construct parallel corpora for research in multilingual medical NLP.
The structured metadata of the corpus provide ample opportunities for future research. For instance, the corpus can be used as a resource for evidence-based medicine summarization, as it contains mappings from literature references to recommendation statements and evidence levels. As we plan to create future versions of the corpus based on updated guideline versions, the extracted concepts can also be used to track changes in CPGs, like the emergence of new treatments and other changes in recommended clinical practice. We envision to combine information extracted from scientific articles, such as study reports, or clinical trial registers with information from CPGs to automatically detect if these CPGs might be outdated given changes in the underlying evidence base.
We make GGPOnc available for researchers under the conditions of a Data Use Agreement. For instructions on how to access the corpus and the human annotated data see: https://www.leitlinienprogramm-onkologie.de/projekte/ggponc-english/. The code to reproduce our experiments is available at: https://github.com/JULIELab/GGPOnc/.
Parts of this work were generously supported by the German Federal Ministry of Research and Education (BMBF) under grants (01ZZ1802H, 01ZZ1803G). We thank all annotators, as well as André Scherag and Danny Ammon from the Jena University Hospital. In addition, we thank all colleagues from the HiGHmed and SMITH consortia for their constant support and valuable input contributing to our joint research.
-  (2011) GRADE guidelines: 3. rating the quality of evidence. Journal of clinical epidemiology 64 (4), pp. 401–406. Cited by: TABLE II.
-  (2020) Clinical concept embeddings learned from massive sources of multimodal medical data. Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing 25, pp. 295–306 (eng). Cited by: §VI.
-  (2017) Semi-automatic mark-up and UMLS annotation of clinical guidelines.. Studies in health technology and informatics 245, pp. 294–297 (eng). Note: Adapted cTakes to detect German UMLS concepts in clinical guidelines. The approach has been evaluated on a single breast cancer guideline. Cited by: §II.
-  (2004-01) The unified medical language system (umls): integrating biomedical terminology. Nucleic Acids Research 32, pp. D267–D270. Cited by: §III-B.
-  (2007) Automatically restructuring practice guidelines using the GEM DTD. In Biological, translational, and clinical language processing, Prague, pp. 113–120. Cited by: §II.
-  (2013) Identifying pathological findings in German radiology reports using a syntacto-semantic parsing approach. In BioNLP 2013 — Proceedings of the Workshop on Biomedical Natural Language Processing @ ACL. Sofia, Bulgaria, pp. 27–35. Cited by: TABLE I.
-  (2016) Negation detection in clinical reports written in German. In BioTxtM 2016 — Proceedings of the 5th Workshop on Workshop on Building and Evaluating Resources for Biomedical Text Mining @ COLING 2016. Osaka, Japan, December 12, 2016, pp. 115–124. Cited by: TABLE I.
-  (2019) A novel NLP-FUZZY system prototype for information extraction from medical guidelines. In 2019 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 1025–1030. Note: Extracted recommendations from PDF versions of 45 guidelines with 1020 recommendations.
-  Used LSTMs to extract ”Action takers”, ”symptoms”, ”actions” and ”purposes”.
-  Then use fuzzy rules to recognize recommendations and predict the grade of recommendation. Cited by: §II.
-  (2012) Information extraction from unstructured electronic health records and integration into a data warehouse. In INFORMATIK 2012: Was bewegt uns in der/die Zukunft? Proceedings der 42. Jahrestagung der Gesellschaft für Informatik e.V. (GI). Braunschweig, Germany, September 16-21, 2012, GI-Edition - Lecture Notes in Informatics (LNI), pp. 1237–1251. Cited by: TABLE I.
-  (2017) Formalizing clinical practice guideline for clinical decision support systems. Health Informatics Journal 23 (2), pp. 146–156. Note: use a rule-based approach to detect procedures and drug recommendations using a subset of the YGRC corpus. Cited by: §II.
-  (2020) German guideline program in oncology. Note: https://www.leitlinienprogramm-onkologie.de/english-language/Accessed: 2020-03-11 Cited by: §I.
-  (2008) Syntactical negation detection in clinical practice guidelines. Studies in health technology and informatics 136, pp. 187–192 (eng). Note: Detect negation triggers and negated concepts using a rule-based approach. Developed using 4 guidelines in English and evaluated on 14 others. Cited by: §II.
-  (23-28) UIMA-based jcore 2.0 goes github and maven central ― state-of-the-art software resource engineering and distribution of nlp pipelines. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, pp. 2502–2509 (english). Cited by: §III-B.
-  (2018) 3000PA: Towards a national reference corpus of German clinical language. In MIE 2018 — Proceedings of the 29th Medical Informatics in Europe Conference. Gothenburg, Sweden, April 23-25, 2018, pp. 26–30. Cited by: TABLE I, §IV-B.
-  (2001) A new system for grading recommendations in evidence based guidelines. Bmj 323 (7308), pp. 334–336. Cited by: TABLE II.
-  (2015) Jufit: a configurable rule engine for filtering and generating new multilingual Umls terms. In AMIA 2015, pp. 604–610. Cited by: §III-B.
-  (2017) Extracting condition-action statements in medical guidelines. In AMIA 2017, American Medical Informatics Association Annual Symposium, Washington, DC, USA, November 4-8, Cited by: §II.
-  (2005) Agreement, the f-measure, and reliability in information retrieval. JAMIA 12 (3), pp. 296–298. Cited by: §IV-C.
-  (2009) The Yale Guideline Recommendation Corpus: a representative sample of the knowledge content of guidelines.. International journal of medical informatics 78 (5), pp. 354–363 (eng). Note: Sample of 1275 guideline recommendations extracted from NGC. The work revealed inconstencies in writing style and reporting of recommendations strength. Cited by: §II.
-  (2016) MIMIC-III, a freely accessible critical care database. Scientific Data 3, pp. #160035. Cited by: §I.
-  (2010) Identifying treatment activities for modelling computer-interpretable clinical practice guidelines. In International Workshop on Knowledge Representation for Health Care, pp. 114–125. Note: Identify actitivies using UMLS Semantic network and manually defined patterns. Evaluated on ”Management of labor” guideline. Cited by: §II.
-  (2019) Overview of the CLEF eHealth evaluation lab 2019. In International Conference of the Cross-Language Evaluation Forum for European Languages, pp. 322–339. Cited by: §I.
-  (2019) Annotating German clinical documents for de-identification. MedInfo 2019, pp. 203–207. Cited by: §IV-C.
-  (2019) Knowledge-based best of breed approach for automated detection of clinical events based on german free text digital hospital discharge letters. PloS one 14 (11). Cited by: TABLE I.
-  (2017) Semi-automatic terminology generation for information extraction from German chest X-ray reports. In Proceedings of the 62nd Annual Meeting of the German Association of Medical Informatics, Biometry and Epidemiology (GMDS e.V.). Oldenburg, pp. 80–84. Cited by: TABLE I.
-  (2016) Unsupervised abbreviation detection in clinical narratives. In ClinicalNLP 2016 — Proceedings of the Clinical Natural Language Processing Workshop @ COLING 2016. Osaka, Japan, December 11, 2016, pp. 91–98. Cited by: TABLE I.
-  (2016) Overlap in drug-disease associations between clinical practice guidelines and drug structured product label indications.. Journal of biomedical semantics 7, pp. 37 (eng). Note: Identify drug-disease relationships via named entity recognition using a corpus of 377 NGC guideline summaries. The extracted relations are compared to structured drug product labels to assess their overlap. Cited by: §II.
-  (2015) Automating identification of multiple chronic conditions in clinical practice Guidelines.. AMIA Joint Summits on Translational Science proceedings, pp. 456–460. Note: Identify comorbid conditions by analyzing pairs of co-occuring conditions, using a corpus of 268 NGC guideline summaries. Cited by: §II.
-  (2018) Sharing copies of synthetic clinical corpora without physical distribution: a case study to get around IPRs and privacy constraints featuring the German JSynCC corpus. In LREC 2018 — Proceedings of the 11th International Conference on Language Resources and Evaluation. Miyazaki, Japan, May 7-12, 2018, pp. 1259–1266. Cited by: TABLE I, §I, §IV-B.
-  (2016) A corpus of german clinical reports for ICD and OPS-based language modeling. In CLAW 2016 — Proceedings of the 6th Workshop on Controlled Language Applications @ LREC 2016, pp. 20–23. Cited by: TABLE I.
-  (2020) An evolutionary approach to the annotation of discharge summaries. In Studies in Health Technology and Informatics—Volume 270: Digital Personalized Health and Medicine, pp. 28 – 32. Cited by: §IV-C.
-  (2011) OCEBM levels of evidence working group. the oxford 2011 levels of evidence. Oxford Centre for Evidence-Based Medicine. Note: http://www.cebm.net/index.aspx?o=5653 Cited by: TABLE II.
-  (2013) Computer-interpretable clinical guidelines: a methodological review. JBI 46 (4), pp. 744–763. Cited by: §II.
-  (23-28) A corpus of clinical practice guidelines annotated with the importance of recommendations. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, pp. 1724–1731 (english). Cited by: §II, §IV-A.
-  (2016) A fine-grained corpus annotation schema of German nephrology records. In ClinicalNLP 2016 — Proceedings of the Clinical Natural Language Processing Workshop @ COLING 2016. Osaka, Japan, December 11, 2016, pp. 69–77. Cited by: TABLE I.
-  (2015) The medical knowledge cockpit: real-time analysis of big medical data enabling precision medicine. In 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 770–775. Cited by: §VI.
-  (2007) Extraction and use of linguistic patterns for modelling medical guidelines. Artificial Intelligence in Medicine 39 (2), pp. 137–149. Note: extraction and instatiation of linguistic templates for guideline formalization, evaluated on a CBO guideline for treatment of breast cancer. Cited by: §II.
-  (2019-06-01) Onkologische Leitlinien – Herausforderungen und zukünftige Entwicklungen. Forum 34 (3), pp. 277–283. Cited by: §I, §VI.
-  (2012) Brat: A Web-based tool for NLP-assisted text annotation. In EACL 2012 — Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics: Demonstrations. Avignon, France, April 25-26, 2012, pp. 102–107. Cited by: §IV-C.
-  (2011-09) WikiWarsDE: a german corpus of narratives annotated with temporal expressions. In Proceedings of the Conference of the German Society for Computational Linguistics and Language Technology (GSCL), pp. 129–134. Cited by: §IV-B.
-  (2018-05) KRAUTS: a German temporally annotated news corpus. In LREC 2018 — 11th International Conference on Language Resources and Evaluation, , pp. 536–540. Cited by: §IV-B.
-  (2015) Fine-grained information extraction from German transthoracic echocardiography reports. BMC Medical Informatics and Decision Making 15, pp. #91. Cited by: TABLE I.
-  (2011) 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. JAMIA 18 (5), pp. 552–556. Cited by: §I.
Identifying condition-action sentences using a heuristic-based information extraction method. In Process Support and Knowledge Representation in Health Care, pp. 26–38. Cited by: §II.
-  (2004) Really, is medical sublanguage that different? Experimental counter-evidence from tagging medical and newspaper corpora. In Proceedings of the 11th World Congress on Medical Informatics, Studies in Health Technology and Informatics, Amsterdam, pp. 560–564. Cited by: TABLE I, §III-B.
-  (2009) High-performance gene name normalization with GeNo. Bioinformatics 25 (6), pp. 815–821. External Links: Cited by: §III-B.
-  (2017) Towards semantic modeling of contradictions and disagreements: a case study of medical guidelines. In IWCS 2017 — Proceedings of the 12th International Conference on Computational Semantics. Montpellier, France 19-22 September 2017, C. Gardent and C. Retoré (Eds.), Vol. 2: Short Papers, pp. #43. Cited by: §II.
-  (2013) Automatic information extraction for computerized clinical guideline.. Studies in health technology and informatics 192, pp. 1023 (eng). Note: use a rule-based approach to extract patient states and actions, evaluated on a single guideline from which 177 recommendation have been extraced. Cited by: §II.