Sparse Binary Transformers for Multivariate Time Series Modeling

by   Matt Gorbett, et al.

Compressed Neural Networks have the potential to enable deep learning across new applications and smaller computational environments. However, understanding the range of learning tasks in which such models can succeed is not well studied. In this work, we apply sparse and binary-weighted Transformers to multivariate time series problems, showing that the lightweight models achieve accuracy comparable to that of dense floating-point Transformers of the same structure. Our model achieves favorable results across three time series learning tasks: classification, anomaly detection, and single-step forecasting. Additionally, to reduce the computational complexity of the attention mechanism, we apply two modifications, which show little to no decline in model performance: 1) in the classification task, we apply a fixed mask to the query, key, and value activations, and 2) for forecasting and anomaly detection, which rely on predicting outputs at a single point in time, we propose an attention mask to allow computation only at the current time step. Together, each compression technique and attention modification substantially reduces the number of non-zero operations necessary in the Transformer. We measure the computational savings of our approach over a range of metrics including parameter count, bit size, and floating point operation (FLOPs) count, showing up to a 53x reduction in storage size and up to 10.5x reduction in FLOPs.


page 1

page 2

page 3

page 4


GluonTS: Probabilistic Time Series Models in Python

We introduce Gluon Time Series (GluonTS)[<>], a...

Transformers in Time Series: A Survey

Transformers have achieved superior performances in many tasks in natura...

Rethinking Attention Mechanism in Time Series Classification

Attention-based models have been widely used in many areas, such as comp...

Erasing-based lossless compression method for streaming floating-point time series

There are a prohibitively large number of floating-point time series dat...

TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting

Transformers have gained popularity in time series forecasting for their...

Focusing on What is Relevant: Time-Series Learning and Understanding using Attention

This paper is a contribution towards interpretability of the deep learni...

Please sign up or login with your details

Forgot password? Click here to reset