research
∙
09/20/2023
Kosmos-2.5: A Multimodal Literate Model
We present Kosmos-2.5, a multimodal literate model for machine reading o...
share
research
∙
09/18/2023
Adapting Large Language Models via Reading Comprehension
We explore how continued pre-training on domain-specific corpora influen...
research
∙
09/03/2023
LogGPT: Exploring ChatGPT for Log-Based Anomaly Detection
The increasing volume of log data produced by software-intensive systems...
research
∙
07/31/2023
Scaling Sentence Embeddings with Large Language Models
Large language models (LLMs) have recently garnered significant interest...
research
∙
07/17/2023
Retentive Network: A Successor to Transformer for Large Language Models
In this work, we propose Retentive Network (RetNet) as a foundation arch...
research
∙
07/05/2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Scaling sequence length has become a critical demand in the era of large...
research
∙
06/26/2023
Kosmos-2: Grounding Multimodal Large Language Models to the World
We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enablin...
research
∙
05/31/2023
Learning Music Sequence Representation from Text Supervision
Music representation learning is notoriously difficult for its complex h...
research
∙
05/16/2023
Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
Recent studies have shown that dual encoder models trained with the sent...
research
∙
05/06/2023
Pre-training Language Model as a Multi-perspective Course Learner
ELECTRA, the generator-discriminator pre-training framework, has achieve...
research
∙
03/21/2023
LogQA: Question Answering in Unstructured Logs
Modern systems produce a large volume of logs to record run-time status ...
research
∙
03/15/2023
UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Large Language Models (LLMs) are popular for their impressive abilities,...
research
∙
02/27/2023
Language Is Not All You Need: Aligning Perception with Language Models
A big convergence of language, multimodal perception, action, and world ...
research
∙
12/20/2022
A Length-Extrapolatable Transformer
Position modeling plays a critical role in Transformers. In this paper, ...
research
∙
12/20/2022
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator
Pre-trained models have achieved remarkable success in natural language ...
research
∙
11/23/2022
TorchScale: Transformers at Scale
Large Transformers have achieved state-of-the-art performance across man...
research
∙
10/26/2022
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
In this paper, we elaborate upon recipes for building multilingual repre...
research
∙
10/13/2022
CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation
Named entity recognition (NER) suffers from the scarcity of annotated tr...
research
∙
10/12/2022
Foundation Transformers
A big convergence of model architectures across language, vision, speech...
research
∙
07/19/2022
MoEC: Mixture of Expert Clusters
Sparsely Mixture of Experts (MoE) has received great interest due to its...
research
∙
06/13/2022
Language Models are General-Purpose Interfaces
Foundation models have received much attention due to their effectivenes...
research
∙
06/01/2022
Task-Specific Expert Pruning for Sparse Mixture-of-Experts
The sparse Mixture-of-Experts (MoE) model is powerful for large-scale pr...
research
∙
06/01/2022
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
As more and more pre-trained language models adopt on-cloud deployment, ...
research
∙
04/20/2022
On the Representation Collapse of Sparse Mixture of Experts
Sparse mixture of experts provides larger model capacity while requiring...
research
∙
03/01/2022
DeepNet: Scaling Transformers to 1,000 Layers
In this paper, we propose a simple yet effective method to stabilize ext...
research
∙
01/15/2022
Kformer: Knowledge Injection in Transformer Feed-Forward Layers
Knowledge-Enhanced Model have developed a diverse set of techniques for ...
research
∙
01/12/2022
PromptBERT: Improving BERT Sentence Embeddings with Prompts
The poor performance of the original BERT for sentence semantic similari...
research
∙
11/03/2021
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
This report describes Microsoft's machine translation systems for the WM...
research
∙
10/21/2021
Improving Non-autoregressive Generation with Mixup Training
While pre-trained language models have achieved great success on various...
research
∙
09/15/2021
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Compared to monolingual models, cross-lingual models usually require a m...
research
∙
06/30/2021
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
In this paper, we introduce ELECTRA-style tasks to cross-lingual languag...
research
∙
06/25/2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
While pretrained encoders have achieved success in various natural langu...
research
∙
06/25/2021
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Large pre-trained models have achieved great success in many natural lan...
research
∙
06/15/2021
Consistency Regularization for Cross-Lingual Fine-Tuning
Fine-tuning pre-trained cross-lingual language models can transfer task-...
research
∙
06/11/2021
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
The cross-lingual language models are typically pretrained with masked l...
research
∙
12/31/2020
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
We generalize deep self-attention distillation in MiniLM (Wang et al., 2...
research
∙
09/24/2020
Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths
Commonsense explanation generation aims to empower the machine's sense-m...
research
∙
09/24/2020
Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph
Despite the success of generative pre-trained language models on a serie...
research
∙
06/01/2020
DocBank: A Benchmark Dataset for Document Layout Analysis
Document layout analysis usually relies on computer vision models to und...
research
∙
12/31/2019
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Pre-training techniques have been verified successfully in a variety of ...
research
∙
03/05/2019
TableBank: Table Benchmark for Image-based Table Detection and Recognition
We present TableBank, a new image-based table detection and recognition ...
research
∙
09/30/2018
Text Morphing
In this paper, we introduce a novel natural language generation task, te...
research
∙
09/12/2018
Neural Melody Composition from Lyrics
In this paper, we study a novel task that learns to compose music from n...
research
∙
07/06/2018
Neural Document Summarization by Jointly Learning to Score and Select Sentences
Sentence scoring and sentence selection are two main steps in extractive...
research
∙
06/21/2018
Dictionary-Guided Editing Networks for Paraphrase Generation
An intuitive way for a human to write paraphrase sentences is to replace...
research
∙
06/19/2018