nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources

09/05/2023

∙

State-of-the-art language models like T5 have revolutionized the NLP landscape, but their computational demands hinder a large portion of the research community. To address this challenge, we present nanoT5, a specially-optimized PyTorch framework for efficient pre-training and fine-tuning of T5 models. Drawing on insights from optimizer differences and prioritizing efficiency, nanoT5 allows a T5-Base model to be pre-trained on a single GPU in just 16 hours, without any loss in performance. With the introduction of this open-source framework, we hope to widen the accessibility to language modelling research and cater to the community's demand for more user-friendly T5 (Encoder-Decoder) implementations. Our contributions, including configurations, codebase, software/hardware insights, and pre-trained models, are available to the public, aiming to strike a balance between research accessibility and resource constraints in NLP.

READ FULL TEXT

nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources

EncT5: Fine-tuning T5 Encoder for Non-autoregressive Tasks

The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models

UER: An Open-Source Toolkit for Pre-training Models

Typos-aware Bottlenecked Pre-Training for Robust Dense Retrieval

CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models

On the comparability of Pre-trained Language Models

MedAlpaca – An Open-Source Collection of Medical Conversational AI Models and Training Data

nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources

Related Research

EncT5: Fine-tuning T5 Encoder for Non-autoregressive Tasks

The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models

UER: An Open-Source Toolkit for Pre-training Models

Typos-aware Bottlenecked Pre-Training for Robust Dense Retrieval

CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models

On the comparability of Pre-trained Language Models

MedAlpaca – An Open-Source Collection of Medical Conversational AI Models and Training Data