Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC

11/07/2022
by   Ze Chen, et al.
0

This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05 We first explore the impact of different pre-trained language models. Then we adopt data cleaning, data augmentation, and adversarial training strategies to enhance the model generalization and robustness. For further improvement, we integrate POS information and word semantic representation using a Mixture-of-Experts (MoE) approach. The experimental results show that MoE can overcome the feature overuse issue and combine the context, POS, and word semantic features well. Additionally, we use a model ensemble method for the final prediction, which has been proven effective by many research works.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/06/2023

OPD@NL4Opt: An ensemble approach for the NER task of the optimization problem

In this paper, we present an ensemble approach for the NL4Opt competitio...
research
04/18/2022

UMass PCL at SemEval-2022 Task 4: Pre-trained Language Model Ensembles for Detecting Patronizing and Condescending Language

Patronizing and condescending language (PCL) is everywhere, but rarely i...
research
06/24/2022

QAGAN: Adversarial Approach To Learning Domain Invariant Language Features

Training models that are robust to data domain shift has gained an incre...
research
11/10/2022

Impact of Adversarial Training on Robustness and Generalizability of Language Models

Adversarial training is widely acknowledged as the most effective defens...
research
04/04/2021

ReCAM@IITK at SemEval-2021 Task 4: BERT and ALBERT based Ensemble for Abstract Word Prediction

This paper describes our system for Task 4 of SemEval-2021: Reading Comp...
research
09/22/2022

Scope of Pre-trained Language Models for Detecting Conflicting Health Information

An increasing number of people now rely on online platforms to meet thei...
research
04/12/2022

Overlapping Word Removal is All You Need: Revisiting Data Imbalance in Hope Speech Detection

Hope Speech Detection, a task of recognizing positive expressions, has m...

Please sign up or login with your details

Forgot password? Click here to reset