b'Shaohuai Shi'

DeepAI AI Chat

Featured Co-authors

Wei Wang
492 publications
Bo Li
461 publications
Xin Liu
216 publications
Yan Zhang
195 publications
Bin Wang
167 publications
Yu Sun
127 publications
Bo Han
121 publications
Ge Li
114 publications
Peng Liu
107 publications
Zhibo Chen
106 publications
Yue Yu
98 publications

research

∙ 09/03/2023

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs

The rapid growth of memory and computation requirements of large languag...

0 Zhenheng Tang, et al. ∙

research

∙ 08/07/2023

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning

The low-rank adaptation (LoRA) method can largely reduce the amount of t...

0 Longteng Zhang, et al. ∙

research

∙ 08/04/2023

Eva: A General Vectorized Approximation Framework for Second-order Optimization

Second-order optimization algorithms exhibit excellent convergence prope...

0 Lin Zhang, et al. ∙

research

∙ 07/04/2023

A Generic Multi-Player Transformation Algorithm for Solving Large-Scale Zero-Sum Extensive-Form Adversarial Team Games

Many recent practical and theoretical breakthroughs focus on adversarial...

0 Chen Qiu, et al. ∙

research

∙ 06/15/2023

Evaluation and Optimization of Gradient Compression for Distributed Deep Learning

To accelerate distributed training, many gradient compression methods ha...

0 Lin Zhang, et al. ∙

research

∙ 03/03/2023

FedML Parrot: A Scalable Federated Learning System via Heterogeneity-aware Scheduling on Sequential and Hierarchical Training

Federated Learning (FL) enables collaborations among clients for train m...

2 Zhenheng Tang, et al. ∙

research

∙ 02/24/2023

Decoupling the All-Reduce Primitive for Accelerating Distributed Deep Learning

Communication scheduling has been shown to be effective in accelerating ...

0 Lin Zhang, et al. ∙

research

∙ 11/30/2022

An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning

To enable the pre-trained models to be fine-tuned with local data on edg...

0 Shaohuai Shi, et al. ∙

research

∙ 07/20/2022

EASNet: Searching Elastic and Accurate Network Architecture for Stereo Matching

Recent advanced studies have spent considerable human efforts on optimiz...

0 Qiang Wang, et al. ∙

research

∙ 06/30/2022

Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning

The second-order optimization methods, notably the D-KFAC (Distributed K...

0 Lin Zhang, et al. ∙

research

∙ 06/06/2022

Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning

In federated learning (FL), model performance typically suffers from cli...

0 Zhenheng Tang, et al. ∙

research

∙ 05/19/2022

Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters

The ever-growing model size and scale of compute have attracted increasi...

8 Yang Xiang, et al. ∙

research

∙ 10/06/2021

FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks

Deep neural networks (DNNs) have achieved great success in the area of c...

9 Qiang Wang, et al. ∙

research

∙ 07/14/2021

Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks

Distributed training with synchronous stochastic gradient descent (SGD) ...

4 Shaohuai Shi, et al. ∙

research

∙ 01/14/2021

Automated Model Design and Benchmarking of 3D Deep Learning Models for COVID-19 Detection with Chest CT Scans

The COVID-19 pandemic has spread globally for several months. Because it...

0 Xin He, et al. ∙

research

∙ 10/20/2020

Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters

Distributed training techniques have been widely deployed in large-scale...

0 Shaohuai Shi, et al. ∙

research

∙ 05/29/2020

Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Format

Multiplication of a sparse matrix to a dense matrix (SpDM) is widely use...

0 Shaohuai Shi, et al. ∙

research

∙ 05/27/2020

Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges

In recent years, distributed deep learning techniques are widely deploye...

0 Shaohuai Shi, et al. ∙

research

∙ 03/24/2020

FADNet: A Fast and Accurate Network for Disparity Estimation

Deep neural networks (DNNs) have achieved great success in the area of c...

13 Qiang Wang, et al. ∙

research

∙ 03/10/2020

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

Distributed deep learning becomes very common to reduce the overall trai...

0 Zhenheng Tang, et al. ∙

research

∙ 02/24/2020

Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs

Distributed Deep Learning (DDL) has rapidly grown its popularity since i...

0 Qiang Wang, et al. ∙

research

∙ 02/22/2020

Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection

Distributed learning techniques such as federated learning have enabled ...

0 Zhenheng Tang, et al. ∙

research

∙ 12/18/2019

MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning

Distributed synchronous stochastic gradient descent has been widely used...

0 Shaohuai Shi, et al. ∙

research

∙ 11/20/2019

Understanding Top-k Sparsification in Distributed Deep Learning

Distributed stochastic gradient descent (SGD) algorithms are widely depl...

0 Shaohuai Shi, et al. ∙

research

∙ 11/20/2019

Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees

To reduce the long training time of large deep neural network (DNN) mode...

0 Shaohuai Shi, et al. ∙

research

∙ 11/20/2019

Computer-Aided Clinical Skin Disease Diagnosis Using CNN and Object Detection Models

Skin disease is one of the most common types of human diseases, which ma...

18 Xin He, et al. ∙

research

∙ 09/15/2019

Performance and Power Evaluation of AI Accelerators for Training Deep Learning Models

Deep neural networks (DNNs) have become widely used in many AI applicati...

0 Yuxin Wang, et al. ∙

research

∙ 01/14/2019

A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks

Distributed synchronous stochastic gradient descent (S-SGD) with data pa...

0 Shaohuai Shi, et al. ∙

research

∙ 11/27/2018

MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms

Distributed synchronous stochastic gradient descent has been widely used...

0 Shaohuai Shi, et al. ∙

research

∙ 07/30/2018

Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Synchronized stochastic gradient descent (SGD) optimizers with data para...

0 Xianyan Jia, et al. ∙

research

∙ 05/10/2018

Modeling and Evaluation of Synchronous Stochastic Gradient Descent in Distributed Deep Learning on Multiple GPUs

With huge amounts of training data, deep learning has made great breakth...

0 Shaohuai Shi, et al. ∙

research

∙ 11/16/2017

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Deep learning frameworks have been widely deployed on GPU servers for de...

0 Shaohuai Shi, et al. ∙

research

∙ 11/09/2017

Performance Evaluation of Deep Learning Tools in Docker Containers

With the success of deep learning techniques in a broad range of applica...

0 Pengfei Xu, et al. ∙

Are you a researcher?
Expose your work
to one of the largest
A.I. communities in the world