On the Usability of Transformers-based models for a French Question-Answering task

07/19/2022
by   Oralie Cattan, et al.
0

For many tasks, state-of-the-art results have been achieved with Transformer-based architectures, resulting in a paradigmatic shift in practices from the use of task-specific architectures to the fine-tuning of pre-trained language models. The ongoing trend consists in training models with an ever-increasing amount of data and parameters, which requires considerable resources. It leads to a strong search to improve resource efficiency based on algorithmic and hardware improvements evaluated only for English. This raises questions about their usability when applied to small-scale learning problems, for which a limited amount of training data is available, especially for under-resourced languages tasks. The lack of appropriately sized corpora is a hindrance to applying data-driven and transfer learning-based approaches with strong instability cases. In this paper, we establish a state-of-the-art of the efforts dedicated to the usability of Transformer-based models and propose to evaluate these improvements on the question-answering performances of French language which have few resources. We address the instability relating to data scarcity by investigating various training strategies with data augmentation, hyperparameters optimization and cross-lingual transfer. We also introduce a new compact model for French FrALBERT which proves to be competitive in low-resource settings.

READ FULL TEXT
research
04/12/2022

MuCoT: Multilingual Contrastive Training for Question-Answering in Low-resource Languages

Accuracy of English-language Question Answering (QA) systems has improve...
research
01/04/2021

Transformers and Transfer Learning for Improving Portuguese Semantic Role Labeling

Semantic Role Labeling (SRL) is a core Natural Language Processing task....
research
07/01/2023

Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin

Developing effective spoken language processing systems for low-resource...
research
10/08/2020

On the importance of pre-training data volume for compact language models

Recent advances in language modeling have led to computationally intensi...
research
04/05/2022

Towards Best Practices for Training Multilingual Dense Retrieval Models

Dense retrieval models using a transformer-based bi-encoder design have ...
research
08/02/2021

ConveRT for FAQ Answering

Knowledgeable FAQ chatbots are a valuable resource to any organization. ...

Please sign up or login with your details

Forgot password? Click here to reset