Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modeling

04/11/2021
by   Aaron Mueller, et al.
0

Neural topic models can augment or replace bag-of-words inputs with the learned representations of deep pre-trained transformer-based word prediction models. One added benefit when using representations from multilingual models is that they facilitate zero-shot polylingual topic modeling. However, while it has been widely observed that pre-trained embeddings should be fine-tuned to a given task, it is not immediately clear what supervision should look like for an unsupervised task such as topic modeling. Thus, we propose several methods for fine-tuning encoders to improve both monolingual and zero-shot polylingual neural topic modeling. We consider fine-tuning on auxiliary tasks, constructing a new topic classification task, integrating the topic classification objective directly into topic model training, and continued pre-training. We find that fine-tuning encoder representations on topic classification and integrating the topic classification task directly into topic modeling improves topic quality, and that fine-tuning encoder representations on any task is the most important factor for facilitating cross-lingual transfer.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 14

07/21/2021

Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer

Multilingual pre-trained contextual embedding models (Devlin et al., 201...
09/04/2021

Robust fine-tuning of zero-shot models

Large pre-trained models such as CLIP offer consistent accuracy across a...
11/15/2021

LiT: Zero-Shot Transfer with Locked-image Text Tuning

This paper presents contrastive-tuning, a simple method employing contra...
06/17/2019

Exploiting Unsupervised Pre-training and Automated Feature Engineering for Low-resource Hate Speech Detection in Polish

This paper presents our contribution to PolEval 2019 Task 6: Hate speech...
12/21/2021

On Cross-Lingual Retrieval with Multilingual Text Encoders

In this work we present a systematic empirical study focused on the suit...
10/14/2021

Composable Sparse Fine-Tuning for Cross-Lingual Transfer

Fine-tuning all parameters of a pre-trained model has become the mainstr...
12/31/2019

oLMpics – On what Language Model Pre-training Captures

Recent success of pre-trained language models (LMs) has spurred widespre...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.