research
∙
09/20/2023
Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2.5, a multimodal literate model for machine reading o...
share
research
∙
09/18/2023
Adapting Large Language Models via Reading Comprehension
We explore how continued pre-training on domain-specific corpora influen...
research
∙
09/11/2023
Large Language Model for Science: A Study on P vs. NP
In this work, we use large language models (LLMs) to augment and acceler...
research
∙
08/24/2023
WavMark: Watermarking for Audio Generation
Recent breakthroughs in zero-shot voice synthesis have enabled imitating...
research
∙
07/17/2023
Retentive Network: A Successor to Transformer for Large Language Models
In this work, we propose Retentive Network (RetNet) as a foundation arch...
research
∙
07/13/2023
In-context Autoencoder for Context Compression in a Large Language Model
We propose the In-context Autoencoder (ICAE) for context compression in ...
research
∙
07/11/2023
Unleashing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration
Human intelligence thrives on the concept of cognitive synergy, where co...
research
∙
07/05/2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large...
research
∙
06/26/2023
Kosmos-2: Grounding Multimodal Large Language Models to the World
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enablin...
research
∙
06/14/2023
Knowledge Distillation of Large Language Models
Knowledge Distillation (KD) is a promising technique for reducing the hi...
research
∙
06/12/2023
Augmenting Language Models with Long-Term Memory
Existing large language models (LLMs) can only afford fix-sized inputs d...
research
∙
05/24/2023
Not All Metrics Are Guilty: Improving NLG Evaluation with LLM Paraphrasing
Most research about natural language generation (NLG) relies on evaluati...
research
∙
05/23/2023
One-stop Training of Multiple Capacity Models for Multilingual Machine Translation
Training models with varying capacities can be advantageous for deployin...
research
∙
05/18/2023
On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation
While multilingual neural machine translation has achieved great success...
research
∙
05/16/2023
Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
Recent studies have shown that dual encoder models trained with the sent...
research
∙
05/16/2023
Pre-Training to Learn in Context
In-context learning, where pre-trained language models learn to perform ...
research
∙
05/11/2023
Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
Large language models (LLMs) demonstrate impressive multilingual capabil...
research
∙
05/11/2023
Chain-of-Dictionary Prompting Elicits Translation in Large Language Models
Large language models (LLMs) have shown surprisingly good performance in...
research
∙
05/06/2023
Pre-training Language Model as a Multi-perspective Course Learner
ELECTRA, the generator-discriminator pre-training framework, has achieve...
research
∙
04/06/2023
On the Pareto Front of Multilingual Neural Machine Translation
In this work, we study how the generalization performance of a given dir...
research
∙
03/15/2023
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Large Language Models (LLMs) are popular for their impressive abilities,...
research
∙
03/02/2023
Semiparametric Language Models Are Scalable Continual Learners
Semiparametric language models (LMs) have shown promise in continuously ...
research
∙
03/01/2023
Are More Layers Beneficial to Graph Transformers?
Despite that going deep has proven successful in many neural architectur...
research
∙
02/28/2023
Generic-to-Specific Distillation of Masked Autoencoders
Large vision Transformers (ViTs) driven by self-supervised pre-training ...
research
∙
02/27/2023
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world ...
research
∙
01/17/2023
HanoiT: Enhancing Context-aware Translation via Selective Context
Context-aware neural machine translation aims to use the document-level ...
research
∙
01/05/2023
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
We introduce a language modeling approach for text to speech synthesis (...
research
∙
12/21/2022
Language Models as Inductive Reasoners
Inductive reasoning is a core component of human intelligence. In the pa...
research
∙
12/20/2022
Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers
Large pretrained language models have shown surprising In-Context Learni...
research
∙
12/20/2022
A Length-Extrapolatable Transformer
Position modeling plays a critical role in Transformers. In this paper, ...
research
∙
12/20/2022
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator
Pre-trained models have achieved remarkable success in natural language ...
research
∙
12/20/2022
Pay Attention to Your Tone: Introducing a New Dataset for Polite Language Rewrite
We introduce PoliteRewrite – a dataset for polite language rewrite which...
research
∙
12/19/2022
Optimizing Prompts for Text-to-Image Generation
Well-designed prompts can guide text-to-image models to generate amazing...
research
∙
12/18/2022
BEATs: Audio Pre-Training with Acoustic Tokenizers
The massive growth of self-supervised learning (SSL) has been witnessed ...
research
∙
12/15/2022
TRIP: Triangular Document-level Pre-training for Multilingual Language Models
Despite the current success of multilingual pre-training, most prior wor...
research
∙
12/13/2022
Structured Prompting: Scaling In-Context Learning to 1,000 Examples
Large language models have exhibited intriguing in-context learning capa...
research
∙
12/08/2022
Momentum Calibration for Text Generation
The input and output of most text generation tasks can be transformed to...
research
∙
12/01/2022
Extensible Prompts for Language Models
We propose eXtensible Prompt (X-Prompt) for prompting a large language m...
research
∙
11/23/2022
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across man...
research
∙
11/21/2022
VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
Although speech is a simple and effective way for humans to communicate ...
research
∙
11/03/2022
Latent Prompt Tuning for Text Summarization
Prompts with different control signals (e.g., length, keywords, etc.) ca...
research
∙
10/31/2022
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation
Direct speech-to-speech translation (S2ST) is an attractive research top...
research
∙
10/26/2022
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
In this paper, we elaborate upon recipes for building multilingual repre...
research
∙
10/19/2022
A Unified View of Masked Image Modeling
Masked image modeling has demonstrated great potential to eliminate the ...
research
∙
10/17/2022
Non-Contrastive Learning Meets Language-Image Pre-Training
Contrastive language-image pre-training (CLIP) serves as a de-facto stan...
research
∙
10/13/2022
CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation
Named entity recognition (NER) suffers from the scarcity of annotated tr...
research
∙
10/12/2022
Foundation Transformers
A big convergence of model architectures across language, vision, speech...
research
∙
10/07/2022
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
The rapid development of single-modal pre-training has prompted research...
research
∙
10/06/2022
XDoc: Unified Pre-training for Cross-Format Document Understanding
The surge of pre-training has witnessed the rapid development of documen...
research
∙
09/30/2022