Is ChatGPT a Biomedical Expert? – Exploring the Zero-Shot Performance of Current GPT Models in Biomedical Tasks

06/28/2023

∙

We assessed the performance of commercial Large Language Models (LLMs) GPT-3.5-Turbo and GPT-4 on tasks from the 2023 BioASQ challenge. In Task 11b Phase B, which is focused on answer generation, both models demonstrated competitive abilities with leading systems. Remarkably, they achieved this with simple zero-shot learning, grounded with relevant snippets. Even without relevant snippets, their performance was decent, though not on par with the best systems. Interestingly, the older and cheaper GPT-3.5-Turbo system was able to compete with GPT-4 in the grounded Q A setting on factoid and list answers. In Task 11b Phase A, focusing on retrieval, query expansion through zero-shot learning improved performance, but the models fell short compared to other systems. The code needed to rerun these experiments is available through GitHub.

READ FULL TEXT

Is ChatGPT a Biomedical Expert? – Exploring the Zero-Shot Performance of Current GPT Models in Biomedical Tasks

Semantically Grounded Visual Embeddings for Zero-Shot Learning

Are Large Language Models Robust Zero-shot Coreference Resolvers?

GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information

Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Demystifying Prompts in Language Models via Perplexity Estimation

Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models

Medical Coding with Biomedical Transformer Ensembles and Zero/Few-shot Learning

Is ChatGPT a Biomedical Expert? – Exploring the Zero-Shot Performance of Current GPT Models in Biomedical Tasks

Related Research

Semantically Grounded Visual Embeddings for Zero-Shot Learning

Are Large Language Models Robust Zero-shot Coreference Resolvers?

GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information

Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Demystifying Prompts in Language Models via Perplexity Estimation

Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models

Medical Coding with Biomedical Transformer Ensembles and Zero/Few-shot Learning