research
∙
09/20/2023
Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2.5, a multimodal literate model for machine reading o...
share
research
∙
07/05/2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large...
research
∙
06/26/2023
Kosmos-2: Grounding Multimodal Large Language Models to the World
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enablin...
research
∙
02/27/2023
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world ...
research
∙
11/23/2022
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across man...
research
∙
10/12/2022
Foundation Transformers
A big convergence of model architectures across language, vision, speech...
research
∙
09/21/2022
DPCN: Towards Deadline-aware Payment Channel Networks
Payment channel is a class of techniques designed to solve the scalabili...
research
∙
08/22/2022
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is eme...
research
∙
06/13/2022
Language Models are General-Purpose Interfaces
Foundation models have received much attention due to their effectivenes...
research
∙
06/02/2022
VL-BEiT: Generative Vision-Language Pretraining
We introduce a vision-language foundation model called VL-BEiT, which is...
research
∙
03/25/2022
TimeReplayer: Unlocking the Potential of Event Cameras for Video Interpolation
Recording fast motion in a high FPS (frame-per-second) requires expensiv...
research
∙
01/29/2022
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Knowledge distillation (KD) methods compress large models into smaller s...
research
∙
12/16/2021
Distilled Dual-Encoder Model for Vision-Language Understanding
We propose a cross-modal attention distillation framework to train a dua...
research
∙
11/03/2021
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
We present a unified Vision-Language pretrained Model (VLMo) that jointl...
research
∙
10/26/2021
s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning
Pretrained bidirectional Transformers, such as BERT, have achieved signi...
research
∙
06/25/2021
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Large pre-trained models have achieved great success in many natural lan...
research
∙
06/15/2021
Consistency Regularization for Cross-Lingual Fine-Tuning
Fine-tuning pre-trained cross-lingual language models can transfer task-...
research
∙
12/31/2020
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
We generalize deep self-attention distillation in MiniLM (Wang et al., 2...
research
∙
07/15/2020
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
In this work, we formulate cross-lingual language model pre-training as ...
research
∙
05/06/2020
Harvesting and Refining Question-Answer Pairs for Unsupervised QA
Question Answering (QA) has shown great success thanks to the availabili...
research
∙
05/02/2020
Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences
Neuromorphic data, recording frameless spike events, have attracted cons...
research
∙
02/28/2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
We propose to pre-train a unified language model for both autoencoding a...
research
∙
02/25/2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its va...
research
∙
09/23/2019
Cross-Lingual Natural Language Generation via Pre-Training
In this work we focus on transferring supervision signals of natural lan...
research
∙
06/14/2019
Learning to Ask Unanswerable Questions for Machine Reading Comprehension
Machine reading comprehension with unanswerable questions is a challengi...
research
∙
05/08/2019