DeepAI AI Chat
Log In Sign Up

In-training Matrix Factorization for Parameter-frugal Neural Machine Translation

by   Zachary Kaden, et al.

In this paper, we propose the use of in-training matrix factorization to reduce the model size for neural machine translation. Using in-training matrix factorization, parameter matrices may be decomposed into the products of smaller matrices, which can compress large machine translation architectures by vastly reducing the number of learnable parameters. We apply in-training matrix factorization to different layers of standard neural architectures and show that in-training factorization is capable of reducing nearly 50 parameters without any associated loss in BLEU score. Further, we find that in-training matrix factorization is especially powerful on embedding layers, providing a simple and effective method to curtail the number of parameters with minimal impact on model performance, and, at times, an increase in performance.


Fast and Low-Memory Deep Neural Networks Using Binary Matrix Factorization

Despite the outstanding performance of deep neural networks in different...

Factorization tricks for LSTM networks

We present two simple ways of reducing the number of parameters and acce...

Recurrent Stacking of Layers for Compact Neural Machine Translation Models

In Neural Machine Translation (NMT), the most common practice is to stac...

The Incremental Multiresolution Matrix Factorization Algorithm

Multiresolution analysis and matrix factorization are foundational tools...

Prior specification via prior predictive matching: Poisson matrix factorization and beyond

Hyperparameter optimization for machine learning models is typically car...

Efficient NTK using Dimensionality Reduction

Recently, neural tangent kernel (NTK) has been used to explain the dynam...

LU-Net: Invertible Neural Networks Based on Matrix Factorization

LU-Net is a simple and fast architecture for invertible neural networks ...