NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports

by   Mael Jullien, et al.

How can we interpret and retrieve medical evidence to support clinical decisions? Clinical trial reports (CTR) amassed over the years contain indispensable information for the development of personalized medicine. However, it is practically infeasible to manually inspect over 400,000+ clinical trial reports in order to find the best evidence for experimental treatments. Natural Language Inference (NLI) offers a potential solution to this problem, by allowing the scalable computation of textual entailment. However, existing NLI models perform poorly on biomedical corpora, and previously published datasets fail to capture the full complexity of inference over CTRs. In this work, we present a novel resource to advance research on NLI for reasoning on CTRs. The resource includes two main tasks. Firstly, to determine the inference relation between a natural language statement, and a CTR. Secondly, to retrieve supporting facts to justify the predicted relation. We provide NLI4CT, a corpus of 2400 statements and CTRs, annotated for these tasks. Baselines on this corpus expose the limitations of existing NLI models, with 6 state-of-the-art NLI models achieving a maximum F1 score of 0.627. To the best of our knowledge, we are the first to design a task that covers the interpretation of full CTRs. To encourage further work on this challenging dataset, we make the corpus, competition leaderboard, website and code to replicate the baseline experiments available at:


page 1

page 2

page 3

page 4


SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data

This paper describes the results of SemEval 2023 task 7 – Multi-Evidence...

THiFLY Research at SemEval-2023 Task 7: A Multi-granularity System for CTR-based Textual Entailment and Evidence Retrieval

The NLI4CT task aims to entail hypotheses based on Clinical Trial Report...

Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports

With the increasing number of clinical trial reports generated every day...

Inferring Which Medical Treatments Work from Reports of Clinical Trials

How do we know if a particular medical treatment actually works? Ideally...

Evidence Inference 2.0: More Data, Better Models

How do we most effectively treat a disease or condition? Ideally, we cou...

Polish Natural Language Inference and Factivity – an Expert-based Dataset and Benchmarks

Despite recent breakthroughs in Machine Learning for Natural Language Pr...

Validating GAN-BioBERT: A Methodology For Assessing Reporting Trends In Clinical Trials

In the past decade, there has been much discussion about the issue of bi...

Please sign up or login with your details

Forgot password? Click here to reset