WIDER CLOSER: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition

12/07/2022
by   Jun-Yu Ma, et al.
0

Zero-shot cross-lingual named entity recognition (NER) aims at transferring knowledge from annotated and rich-resource data in source languages to unlabeled and lean-resource data in target languages. Existing mainstream methods based on the teacher-student distillation framework ignore the rich and complementary information lying in the intermediate layers of pre-trained language models, and domain-invariant information is easily lost during transfer. In this study, a mixture of short-channel distillers (MSD) method is proposed to fully interact the rich hierarchical information in the teacher model and to transfer knowledge to the student model sufficiently and efficiently. Concretely, a multi-channel distillation framework is designed for sufficient information transfer by aggregating multiple distillers as a mixture. Besides, an unsupervised method adopting parallel domain adaptation is proposed to shorten the channels between the teacher and student models to preserve domain-invariant features. Experiments on four datasets across nine languages demonstrate that the proposed method achieves new state-of-the-art performance on zero-shot cross-lingual NER and shows great generalization and compatibility across languages and fields.

READ FULL TEXT

page 1

page 5

research
04/26/2020

Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learning on Unlabeled Data in Target Language

To better tackle the named entity recognition (NER) problem on languages...
research
01/21/2023

ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition

For named entity recognition (NER) in zero-resource languages, utilizing...
research
11/15/2022

DualNER: A Dual-Teaching framework for Zero-shot Cross-lingual Named Entity Recognition

We present DualNER, a simple and effective framework to make full use of...
research
09/01/2021

Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation

Recent multilingual pre-trained language models have achieved remarkable...
research
05/21/2023

SHINE: Syntax-augmented Hierarchical Interactive Encoder for Zero-shot Cross-lingual Information Extraction

Zero-shot cross-lingual information extraction(IE) aims at constructing ...
research
11/14/2019

Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources

For languages with no annotated resources, transferring knowledge from r...
research
01/30/2020

Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages

Most combinations of NLP tasks and language varieties lack in-domain exa...

Please sign up or login with your details

Forgot password? Click here to reset