research
∙
08/23/2023
D4: Improving LLM Pretraining via Document De-Duplication and Diversification
Over recent years, an increasing amount of compute and data has been pou...
share
research
∙
06/26/2023
Understanding In-Context Learning via Supportive Pretraining Data
In-context learning (ICL) improves language models' performance on a var...
research
∙
05/23/2023
Evaluating end-to-end entity linking on domain-specific knowledge bases: Learning about ancient technologies from museum collections
To study social, economic, and historical questions, researchers in the ...
research
∙
05/12/2023
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
Autoregressive transformers are spectacular models for short sequences b...
research
∙
03/16/2023
SemDeDup: Data-efficient learning at web-scale through semantic deduplication
Progress in machine learning has been driven in large part by massive in...
research
∙
12/22/2022
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Recent work has shown that fine-tuning large pre-trained language models...
research
∙
10/04/2022
Text Characterization Toolkit
In NLP, models are usually evaluated by reporting single-number performa...
research
∙
05/12/2022
Open Vocabulary Extreme Classification Using Generative Models
The extreme multi-label classification (XMC) task aims at tagging conten...
research
∙
05/02/2022
OPT: Open Pre-trained Transformer Language Models
Large language models, which are often trained for hundreds of thousands...
research
∙
12/20/2021