research
∙
05/25/2023
Automated Tensor Model Parallelism with Overlapped Communication for Efficient Foundation Model Training
Deep learning is experiencing a rise in foundation models that are expec...
share
research
∙
06/10/2022
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
Foundation models are becoming the dominant deep learning technologies. ...
research
∙
03/30/2022
DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation
The further development of deep neural networks is hampered by the limit...
research
∙
10/18/2021
EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
Distributed data-parallel training has been widely used for natural lang...
research
∙
10/05/2021
S2 Reducer: High-Performance Sparse Communication to Accelerate Distributed Deep Learning
Distributed stochastic gradient descent (SGD) approach has been widely u...
research
∙
04/13/2021
Hierarchical Adaptive Pooling by Capturing High-order Dependency for Graph Representation Learning
Graph neural networks (GNN) have been proven to be mature enough for han...
research
∙
06/10/2020