Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems

05/04/2021
by   Xiaocong Du, et al.
15

Deep learning recommendation systems at scale have provided remarkable gains through increasing model capacity (i.e. wider and deeper neural networks), but it comes at significant training cost and infrastructure cost. Model pruning is an effective technique to reduce computation overhead for deep neural networks by removing redundant parameters. However, modern recommendation systems are still thirsty for model capacity due to the demand for handling big data. Thus, pruning a recommendation model at scale results in a smaller model capacity and consequently lower accuracy. To reduce computation cost without sacrificing model capacity, we propose a dynamic training scheme, namely alternate model growth and pruning, to alternatively construct and prune weights in the course of training. Our method leverages structured sparsification to reduce computational cost without hurting the model capacity at the end of offline training so that a full-size model is available in the recurring training stage to learn new data in real-time. To the best of our knowledge, this is the first work to provide in-depth experiments and discussion of applying structural dynamics to recommendation systems at scale to reduce training cost. The proposed method is validated with an open-source deep-learning recommendation model (DLRM) and state-of-the-art industrial-scale production models.

READ FULL TEXT
10/16/2020

Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data

Large scale deep learning provides a tremendous opportunity to improve t...
11/05/2020

CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery

The paper proposes and optimizes a partial recovery training system, CPR...
10/17/2020

Check-N-Run: A Checkpointing System for Training Recommendation Models

Checkpoints play an important role in training recommendation systems at...
01/19/2021

Learnable Embedding Sizes for Recommender Systems

The embedding-based representation learning is commonly used in deep lea...
06/14/2019

A Strategy for Expert Recommendation From Open Data Available on the Lattes Platform

With the increasing volume of data and users of curriculum systems, the ...
01/16/2013

Big Neural Networks Waste Capacity

This article exposes the failure of some big neural networks to leverage...
05/31/2019

Deep Learning Recommendation Model for Personalization and Recommendation Systems

With the advent of deep learning, neural network-based recommendation mo...