Medical SANSformers: Training self-supervised transformers without attention for Electronic Medical Records

08/31/2021
by   Yogesh Kumar, et al.
0

We leverage deep sequential models to tackle the problem of predicting healthcare utilization for patients, which could help governments to better allocate resources for future healthcare use. Specifically, we study the problem of divergent subgroups, wherein the outcome distribution in a smaller subset of the population considerably deviates from that of the general population. The traditional approach for building specialized models for divergent subgroups could be problematic if the size of the subgroup is very small (for example, rare diseases). To address this challenge, we first develop a novel attention-free sequential model, SANSformers, instilled with inductive biases suited for modeling clinical codes in electronic medical records. We then design a task-specific self-supervision objective and demonstrate its effectiveness, particularly in scarce data settings, by pre-training each model on the entire health registry (with close to one million patients) before fine-tuning for downstream tasks on the divergent subgroups. We compare the novel SANSformer architecture with the LSTM and Transformer models using two data sources and a multi-task learning objective that aids healthcare utilization prediction. Empirically, the attention-free SANSformer models perform consistently well across experiments, outperforming the baselines in most cases by at least ∼ 10%. Furthermore, the self-supervised pre-training boosts performance significantly throughout, for example by over ∼ 50% (and as high as 800%) on R^2 score when predicting the number of hospital visits.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/24/2021

Pre-training transformer-based framework on large-scale pediatric claims data for downstream population-specific tasks

The adoption of electronic health records (EHR) has become universal dur...
research
07/21/2022

Unsupervised pre-training of graph transformers on patient population graphs

Pre-training has shown success in different areas of machine learning, s...
research
03/23/2022

Unsupervised Pre-Training on Patient Population Graphs for Patient-Level Predictions

Pre-training has shown success in different areas of machine learning, s...
research
06/02/2019

Pre-training of Graph Augmented Transformers for Medication Recommendation

Medication recommendation is an important healthcare application. It is ...
research
02/03/2020

Learning Contextualized Document Representations for Healthcare Answer Retrieval

We present Contextual Discourse Vectors (CDV), a distributed document re...
research
06/09/2021

Self-Supervised Graph Learning with Hyperbolic Embedding for Temporal Health Event Prediction

Electronic Health Records (EHR) have been heavily used in modern healthc...
research
07/15/2020

Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-based Neural Networks

In this paper we study the problem of predicting clinical diagnoses from...

Please sign up or login with your details

Forgot password? Click here to reset