b'Furu Wei'

research

∙ 09/20/2023

Kosmos-2.5: A Multimodal Literate Model

We present Kosmos-2.5, a multimodal literate model for machine reading o...

0 Tengchao Lv, et al. ∙

research

∙ 09/18/2023

Adapting Large Language Models via Reading Comprehension

We explore how continued pre-training on domain-specific corpora influen...

0 Daixuan Cheng, et al. ∙

research

∙ 09/11/2023

Large Language Model for Science: A Study on P vs. NP

In this work, we use large language models (LLMs) to augment and acceler...

0 Qingxiu Dong, et al. ∙

research

∙ 08/24/2023

WavMark: Watermarking for Audio Generation

Recent breakthroughs in zero-shot voice synthesis have enabled imitating...

0 Guangyu Chen, et al. ∙

research

∙ 07/17/2023

Retentive Network: A Successor to Transformer for Large Language Models

In this work, we propose Retentive Network (RetNet) as a foundation arch...

0 Yutao Sun, et al. ∙

research

∙ 07/13/2023

In-context Autoencoder for Context Compression in a Large Language Model

We propose the In-context Autoencoder (ICAE) for context compression in ...

0 Tao Ge, et al. ∙

research

∙ 07/11/2023

Unleashing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration

Human intelligence thrives on the concept of cognitive synergy, where co...

0 Zhenhailong Wang, et al. ∙

research

∙ 07/05/2023

LongNet: Scaling Transformers to 1,000,000,000 Tokens

Scaling sequence length has become a critical demand in the era of large...

0 Jiayu Ding, et al. ∙

research

∙ 06/26/2023

Kosmos-2: Grounding Multimodal Large Language Models to the World

We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enablin...

0 Zhiliang Peng, et al. ∙

research

∙ 06/14/2023

Knowledge Distillation of Large Language Models

Knowledge Distillation (KD) is a promising technique for reducing the hi...

0 Yuxian Gu, et al. ∙

research

∙ 06/12/2023

Augmenting Language Models with Long-Term Memory

Existing large language models (LLMs) can only afford fix-sized inputs d...

0 Weizhi Wang, et al. ∙

research

∙ 05/24/2023

Not All Metrics Are Guilty: Improving NLG Evaluation with LLM Paraphrasing

Most research about natural language generation (NLG) relies on evaluati...

0 Tianyi Tang, et al. ∙

research

∙ 05/23/2023

One-stop Training of Multiple Capacity Models for Multilingual Machine Translation

Training models with varying capacities can be advantageous for deployin...

0 Lan Jiang, et al. ∙

research

∙ 05/18/2023

On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation

While multilingual neural machine translation has achieved great success...

0 Liang Chen, et al. ∙

research

∙ 05/16/2023

Dual-Alignment Pre-training for Cross-lingual Sentence Embedding

Recent studies have shown that dual encoder models trained with the sent...

0 Ziheng Li, et al. ∙

research

∙ 05/16/2023

Pre-Training to Learn in Context

In-context learning, where pre-trained language models learn to perform ...

0 Yuxian Gu, et al. ∙

research

∙ 05/11/2023

Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting

Large language models (LLMs) demonstrate impressive multilingual capabil...

0 Haoyang Huang, et al. ∙

research

∙ 05/11/2023

Chain-of-Dictionary Prompting Elicits Translation in Large Language Models

Large language models (LLMs) have shown surprisingly good performance in...

0 Hongyuan Lu, et al. ∙

research

∙ 05/06/2023

Pre-training Language Model as a Multi-perspective Course Learner

ELECTRA, the generator-discriminator pre-training framework, has achieve...

0 Beiduo Chen, et al. ∙

research

∙ 04/06/2023

On the Pareto Front of Multilingual Neural Machine Translation

In this work, we study how the generalization performance of a given dir...

0 Liang Chen, et al. ∙

research

∙ 03/15/2023

UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation

Large Language Models (LLMs) are popular for their impressive abilities,...

0 Daixuan Cheng, et al. ∙

research

∙ 03/02/2023

Semiparametric Language Models Are Scalable Continual Learners

Semiparametric language models (LMs) have shown promise in continuously ...

0 Guangyue Peng, et al. ∙

research

∙ 03/01/2023

Are More Layers Beneficial to Graph Transformers?

Despite that going deep has proven successful in many neural architectur...

0 Haiteng Zhao, et al. ∙

research

∙ 02/28/2023

Generic-to-Specific Distillation of Masked Autoencoders

Large vision Transformers (ViTs) driven by self-supervised pre-training ...

0 Wei Huang, et al. ∙

research

∙ 02/27/2023

Language Is Not All You Need: Aligning Perception with Language Models

A big convergence of language, multimodal perception, action, and world ...

0 Shaohan Huang, et al. ∙

research

∙ 01/17/2023

HanoiT: Enhancing Context-aware Translation via Selective Context

Context-aware neural machine translation aims to use the document-level ...

0 Jian Yang, et al. ∙

research

∙ 01/05/2023

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

We introduce a language modeling approach for text to speech synthesis (...

4 Chengyi Wang, et al. ∙

research

∙ 12/21/2022

Language Models as Inductive Reasoners

Inductive reasoning is a core component of human intelligence. In the pa...

0 Zonglin Yang, et al. ∙

research

∙ 12/20/2022

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers

Large pretrained language models have shown surprising In-Context Learni...

0 Damai Dai, et al. ∙

research

∙ 12/20/2022

A Length-Extrapolatable Transformer

Position modeling plays a critical role in Transformers. In this paper, ...

0 Yutao Sun, et al. ∙

research

∙ 12/20/2022

GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator

Pre-trained models have achieved remarkable success in natural language ...

0 Jian Yang, et al. ∙

research

∙ 12/20/2022

Pay Attention to Your Tone: Introducing a New Dataset for Polite Language Rewrite

We introduce PoliteRewrite – a dataset for polite language rewrite which...

0 Xun Wang, et al. ∙

research

∙ 12/19/2022

Optimizing Prompts for Text-to-Image Generation

Well-designed prompts can guide text-to-image models to generate amazing...

0 Yaru Hao, et al. ∙

research

∙ 12/18/2022

BEATs: Audio Pre-Training with Acoustic Tokenizers

The massive growth of self-supervised learning (SSL) has been witnessed ...

0 Sanyuan Chen, et al. ∙

research

∙ 12/15/2022

TRIP: Triangular Document-level Pre-training for Multilingual Language Models

Despite the current success of multilingual pre-training, most prior wor...

0 Hongyuan Lu, et al. ∙

research

∙ 12/13/2022

Structured Prompting: Scaling In-Context Learning to 1,000 Examples

Large language models have exhibited intriguing in-context learning capa...

0 Yaru Hao, et al. ∙

research

∙ 12/08/2022

Momentum Calibration for Text Generation

The input and output of most text generation tasks can be transformed to...

0 Xingxing Zhang, et al. ∙

research

∙ 12/01/2022

Extensible Prompts for Language Models

We propose eXtensible Prompt (X-Prompt) for prompting a large language m...

0 Tao Ge, et al. ∙

research

∙ 11/23/2022

TorchScale: Transformers at Scale

Large Transformers have achieved state-of-the-art performance across man...

0 Shuming Ma, et al. ∙

research

∙ 11/21/2022

VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning

Although speech is a simple and effective way for humans to communicate ...

0 Qiushi Zhu, et al. ∙

research

∙ 11/03/2022

Latent Prompt Tuning for Text Summarization

Prompts with different control signals (e.g., length, keywords, etc.) ca...

0 Yubo Zhang, et al. ∙

research

∙ 10/31/2022

Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation

Direct speech-to-speech translation (S2ST) is an attractive research top...

0 Kun Wei, et al. ∙

research

∙ 10/26/2022

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

In this paper, we elaborate upon recipes for building multilingual repre...

0 Barun Patra, et al. ∙

research

∙ 10/19/2022

A Unified View of Masked Image Modeling

Masked image modeling has demonstrated great potential to eliminate the ...

0 Zhiliang Peng, et al. ∙

research

∙ 10/17/2022

Non-Contrastive Learning Meets Language-Image Pre-Training

Contrastive language-image pre-training (CLIP) serves as a de-facto stan...

1 Jinghao Zhou, et al. ∙

research

∙ 10/13/2022

CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation

Named entity recognition (NER) suffers from the scarcity of annotated tr...

0 Jian Yang, et al. ∙

research

∙ 10/12/2022

Foundation Transformers

A big convergence of model architectures across language, vision, speech...

26 Hongyu Wang, et al. ∙

research

∙ 10/07/2022

SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training

The rapid development of single-modal pre-training has prompted research...

0 Ziqiang Zhang, et al. ∙

research

∙ 10/06/2022

XDoc: Unified Pre-training for Cross-Format Document Understanding

The surge of pre-training has witnessed the rapid development of documen...

0 Jingye Chen, et al. ∙

research

∙ 09/30/2022

SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

How to boost speech pre-training with textual data is an unsolved proble...

0 Ziqiang Zhang, et al. ∙

Furu Wei

Featured Co-authors