research
∙
07/13/2023
Layerwise Linear Mode Connectivity
In the federated setup one performs an aggregation of separate local mod...
share
research
∙
06/14/2023
Provably Personalized and Robust Federated Learning
Clustering clients with similar objectives and learning a model per clus...
research
∙
06/01/2023
Faster Causal Attention Over Large Sequences Through Sparse Flash Attention
Transformer-based language models have found many diverse applications r...
research
∙
05/30/2023
Shuffle SGD is Always Better than SGD: Improved Analysis of SGD with Arbitrary Data Orders
Stochastic Gradient Descent (SGD) algorithms are widely used in optimizi...
research
∙
05/29/2023
Collaborative Learning via Prediction Consensus
We consider a collaborative learning setting where each agent's goal is ...
research
∙
05/26/2023
Rotational Optimizers: Simple Robust DNN Training
The training dynamics of modern deep neural networks depend on complex i...
research
∙
05/26/2023
Ghost Noise for Regularizing Deep Neural Networks
Batch Normalization (BN) is widely used to stabilize the optimization pr...
research
∙
05/26/2023
Hardware-Efficient Transformer Training via Piecewise Affine Operations
Multiplications are responsible for most of the computational cost invol...
research
∙
05/25/2023
Landmark Attention: Random-Access Infinite Context Length for Transformers
While transformers have shown remarkable success in natural language pro...
research
∙
02/24/2023
Linearization Algorithms for Fully Composite Optimization
In this paper, we study first-order algorithms for solving fully composi...
research
∙
02/23/2023
Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods
We study the widely known Cubic-Newton method in the stochastic setting ...
research
∙
01/05/2023
Beyond spectral gap (extended): The role of the topology in decentralized learning
In data-parallel optimization of machine learning models, workers collab...
research
∙
12/01/2022
Second-order optimization with lazy Hessians
We analyze Newton's method with lazy Hessian updates for solving general...
research
∙
11/19/2022
Accuracy Boosters: Epoch-Driven Mixed-Mantissa Block Floating-Point for DNN Training
The unprecedented growth in DNN model complexity, size and the amount of...
research
∙
11/12/2022
Modular Clinical Decision Support Networks (MoDN) – Updatable, Interpretable, and Portable Predictions for Evolving Clinical Environments
Data-driven Clinical Decision Support Systems (CDSS) have the potential ...
research
∙
06/16/2022
Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning
We study the asynchronous stochastic gradient descent algorithm for dist...
research
∙
06/07/2022
Beyond spectral gap: The role of the topology in decentralized learning
In data-parallel optimization of machine learning models, workers collab...
research
∙
05/30/2022
On Avoiding Local Minima Using Gradient Descent With Large Learning Rates
It has been widely observed in training of neural networks that when app...
research
∙
05/17/2022
SKILL: Structured Knowledge Infusion for Large Language Models
Large language models (LLMs) have demonstrated human-level performance o...
research
∙
04/13/2022
Data-heterogeneity-aware Mixing for Decentralized Learning
Decentralized learning provides an effective framework to train machine ...
research
∙
02/11/2022
Improving Generalization via Uncertainty Driven Perturbations
Recently Shah et al., 2020 pointed out the pitfalls of the simplicity bi...
research
∙
02/09/2022
Agree to Disagree: Diversity through Disagreement for Better Transferability
Gradient-based learning algorithms have an implicit simplicity bias whic...
research
∙
02/03/2022
Characterizing Finding Good Data Orderings for Fast Convergence of Sequential Gradient Methods
While SGD, which samples from the data with replacement is widely studie...
research
∙
02/03/2022
Byzantine-Robust Decentralized Learning via Self-Centered Clipping
In this paper, we study the challenging task of Byzantine-robust decentr...
research
∙
12/16/2021
Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimation
Over-parameterized deep neural networks are able to achieve excellent tr...
research
∙
11/16/2021
Interpreting Language Models Through Knowledge Graph Extraction
Transformer-based language models trained on large text corpora have enj...
research
∙
11/10/2021
Linear Speedup in Personalized Collaborative Learning
Personalization in federated learning can improve the accuracy of a mode...
research
∙
10/25/2021
Optimal Model Averaging: Towards Personalized Collaborative Learning
In federated learning, differences in the data or objectives between the...
research
∙
10/13/2021
WAFFLE: Weighted Averaging for Personalized Federated Learning
In collaborative or federated learning, model personalization can be a v...
research
∙
10/08/2021
RelaySum for Decentralized Deep Learning on Heterogeneous Data
In decentralized machine learning, workers compute model updates on thei...
research
∙
09/06/2021
On Second-order Optimization Methods for Federated Learning
We consider federated learning (FL), where the training data is distribu...
research
∙
08/18/2021
Semantic Perturbations with Normalizing Flows for Improved Generalization
Data augmentation is a widely adopted technique for avoiding overfitting...
research
∙
07/14/2021
IFedAvg: Interpretable Data-Interoperability for Federated Learning
Recently, the ever-growing demand for privacy-oriented machine learning ...
research
∙
06/25/2021
Implicit Gradient Alignment in Distributed and Federated Learning
A major obstacle to achieving global convergence in distributed and fede...
research
∙
06/16/2021
Simultaneous Training of Partially Masked Neural Networks
For deploying deep learning models to lower end devices, it is necessary...
research
∙
06/08/2021
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
The advent of contextual word embeddings – representations of words whic...
research
∙
05/28/2021
Lightweight Cross-Lingual Sentence Representation Learning
Large-scale models for learning fixed-dimensional cross-lingual sentence...
research
∙
04/15/2021
Federated Learning for Malware Detection in IoT Devices
This work investigates the possibilities enabled by federated learning c...
research
∙
03/03/2021
Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates
It has been experimentally observed that the efficiency of distributed t...
research
∙
02/09/2021
Consensus Control for Decentralized Deep Learning
Decentralized training of deep learning models enables on-device learnin...
research
∙
02/09/2021
Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
Decentralized training of deep learning models is a key element for enab...
research
∙
02/05/2021
Exact Optimization of Conformal Predictors via Incremental and Decremental Learning
Conformal Predictors (CP) are wrappers around ML methods, providing erro...
research
∙
12/18/2020
Learning from History for Byzantine Robust Optimization
Byzantine robustness has received significant attention recently given i...
research
∙
11/03/2020
A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!
Decentralized optimization methods enable on-device training of machine ...
research
∙
09/19/2020
Sparse Communication for Training Deep Networks
Synchronous stochastic gradient descent (SGD) is the most common method ...
research
∙
08/08/2020
Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
Federated learning is a challenging optimization problem due to the hete...
research
∙
08/04/2020
PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
Lossy gradient compression has become a practical tool to overcome the c...
research
∙
06/29/2020
Multi-Head Attention: Collaborate Instead of Concatenate
Attention layers are widely used in natural language processing (NLP) an...
research
∙
06/25/2020
Taming GANs with Lookahead
Generative Adversarial Networks are notoriously challenging to train. Th...
research
∙
06/16/2020
