SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification

02/01/2021

∙

In this paper we present our submission for the EACL 2021-Shared Task on Offensive Language Identification in Dravidian languages. Our final system is an ensemble of mBERT and XLM-RoBERTa models which leverage task-adaptive pre-training of multilingual BERT models with a masked language modeling objective. Our system was ranked 1st for Kannada, 2nd for Malayalam and 3rd for Tamil.

READ FULL TEXT

SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification

Improved Self-Supervised Multilingual Speech Representation Learning Combined with Auxiliary Language Information

OPI at SemEval 2023 Task 9: A Simple But Effective Approach to Multilingual Tweet Intimacy Analysis

Multilingual Pre-Trained Transformers and Convolutional NN Classification Models for Technical Domain Identification

Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models

ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models

Towards Structured Dynamic Sparse Pre-Training of BERT

SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification

Related Research

Improved Self-Supervised Multilingual Speech Representation Learning Combined with Auxiliary Language Information

OPI at SemEval 2023 Task 9: A Simple But Effective Approach to Multilingual Tweet Intimacy Analysis

Multilingual Pre-Trained Transformers and Convolutional NN Classification Models for Technical Domain Identification

Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models

ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain

What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models

Towards Structured Dynamic Sparse Pre-Training of BERT