A Family of Pretrained Transformer Language Models for Russian

09/19/2023

∙

Nowadays, Transformer language models (LMs) represent a fundamental component of the NLP research methodologies and applications. However, the development of such models specifically for the Russian language has received little attention. This paper presents a collection of 13 Russian Transformer LMs based on the encoder (ruBERT, ruRoBERTa, ruELECTRA), decoder (ruGPT-3), and encoder-decoder (ruT5, FRED-T5) models in multiple sizes. Access to these models is readily available via the HuggingFace platform. We provide a report of the model architecture design and pretraining, and the results of evaluating their generalization abilities on Russian natural language understanding and generation datasets and benchmarks. By pretraining and releasing these specialized Transformer LMs, we hope to broaden the scope of the NLP research directions and enable the development of industrial solutions for the Russian language.

READ FULL TEXT

A Family of Pretrained Transformer Language Models for Russian

Transformer-based Korean Pretrained Language Models: A Survey on Three Years of Progress

Efficient Long-Text Understanding with Short-Text Models

On the Computational Power of Decoder-Only Transformer Language Models

A PhD Student's Perspective on Research in NLP in the Era of Very Large Language Models

What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization

Task-agnostic Distillation of Encoder-Decoder Language Models

Pretraining on the Test Set Is All You Need

A Family of Pretrained Transformer Language Models for Russian

Related Research

Transformer-based Korean Pretrained Language Models: A Survey on Three Years of Progress

Efficient Long-Text Understanding with Short-Text Models

On the Computational Power of Decoder-Only Transformer Language Models

A PhD Student's Perspective on Research in NLP in the Era of Very Large Language Models

What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization

Task-agnostic Distillation of Encoder-Decoder Language Models

Pretraining on the Test Set Is All You Need