research
∙
05/21/2023
Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers
This paper explores the effectiveness of model-generated signals in impr...
share
research
∙
02/27/2023
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world ...
research
∙
12/20/2022
A Length-Extrapolatable Transformer
Position modeling plays a critical role in Transformers. In this paper, ...
research
∙
11/23/2022
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across man...
research
∙
10/26/2022
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
In this paper, we elaborate upon recipes for building multilingual repre...
research
∙
10/12/2022
Foundation Transformers
A big convergence of model architectures across language, vision, speech...
research
∙
04/20/2022
On the Representation Collapse of Sparse Mixture of Experts
Sparse mixture of experts provides larger model capacity while requiring...
research
∙
04/13/2022
METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals
We present an efficient method of pretraining large-scale autoencoding l...
research
∙
04/07/2022
Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
We present a new framework AMOS that pretrains text encoders with an Adv...
research
∙
01/28/2022
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Pretrained general-purpose language models can achieve state-of-the-art ...
research
∙
11/03/2021
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WM...
research
∙
09/15/2021
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Compared to monolingual models, cross-lingual models usually require a m...
research
∙
06/30/2021
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual languag...
research
∙
06/25/2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural langu...
research
∙
06/15/2021
Consistency Regularization for Cross-Lingual Fine-Tuning
Fine-tuning pre-trained cross-lingual language models can transfer task-...
research
∙
06/04/2021
Language Scaling for Universal Suggested Replies Model
We consider the problem of scaling automated suggested replies for Outlo...
research
∙
02/16/2021
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
We present COCO-LM, a new self-supervised learning framework that pretra...
research
∙
12/31/2020
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Multilingual machine translation enables a single model to translate bet...
research
∙
07/15/2020
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we formulate cross-lingual language model pre-training as ...
research
∙
06/29/2020
Knowledge-Aware Language Model Pretraining
How much knowledge do pretrained language models hold? Recent research o...
research
∙
07/24/2019
Generic Intent Representation in Web Search
This paper presents GEneric iNtent Encoder (GEN Encoder) which learns a ...
research
∙
04/15/2019
An Axiomatic Approach to Regularizing Neural Ranking Models
Axiomatic information retrieval (IR) seeks a set of principle properties...
research
∙
09/23/2018
Towards Language Agnostic Universal Representations
When a bilingual student learns to solve word problems in math, we expec...
research
∙
11/25/2017
Neural Ranking Models with Multiple Document Fields
Deep neural networks have recently shown promise in the ad-hoc retrieval...
research
∙
11/28/2016