research
∙
07/05/2023
Improving Automatic Parallel Training via Balanced Memory Workload Optimization
Transformer models have emerged as the leading approach for achieving st...
share
research
∙
04/08/2023
FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
With the increasing data volume, there is a trend of using large-scale p...
research
∙
03/06/2023
Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent
Recent years have witnessed the unprecedented achievements of large-scal...
research
∙
11/25/2022
Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism
Transformer models have achieved state-of-the-art performance on various...
research
∙
09/27/2022
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Large-scale deep learning models contribute to significant performance i...
research
∙
03/28/2022
HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System
As giant dense models advance quality but require large-scale expensive ...
research
∙
12/29/2021
Dense-to-Sparse Gate for Mixture-of-Experts
Mixture-of-experts (MoE) is becoming popular due to its success in impro...
research
∙
12/14/2021