DeepAI AI Chat
Log In Sign Up

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

by   Md. Tawkat Islam Khondaker, et al.
The University of British Columbia

The recent emergence of ChatGPT has brought a revolutionary change in the landscape of NLP. Although ChatGPT has consistently shown impressive performance on English benchmarks, its exact capabilities on most other languages remain largely unknown. To better understand ChatGPT's capabilities on Arabic, we present a large-scale evaluation of the model on a broad range of Arabic NLP tasks. Namely, we evaluate ChatGPT on 32 diverse natural language understanding and generation tasks on over 60 different datasets. To the best of our knowledge, our work offers the first performance analysis of ChatGPT on Arabic NLP at such a massive scale. Our results show that, despite its success on English benchmarks, ChatGPT trained in-context (few-shot) is consistently outperformed by much smaller dedicated models finetuned on Arabic. These results suggest that there is significant place for improvement for instruction-tuned LLMs such as ChatGPT.


page 11

page 16


A Large and Diverse Arabic Corpus for Language Modeling

Language models (LMs) have introduced a major paradigm shift in Natural ...

ChatGPT for Arabic Grammatical Error Correction

Recently, large language models (LLMs) fine-tuned to follow human instru...

What Makes Good In-Context Examples for GPT-3?

GPT-3 has attracted lots of attention due to its superior performance ac...

A Benchmark Study of Contrastive Learning for Arabic Social Meaning

Contrastive learning (CL) brought significant progress to various NLP ta...

Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models

Large language models (LLMs) have demonstrated impressive performance on...

From Local to Global: Navigating Linguistic Diversity in the African Context

The focus is on critical problems in NLP related to linguistic diversity...

ORCA: A Challenging Benchmark for Arabic Language Understanding

Due to their crucial role in all NLP, several benchmarks have been propo...