Effectively Modeling Time Series with Simple Discrete State Spaces

03/16/2023
by   Michael Zhang, et al.
0

Time series modeling is a well-established problem, which often requires that methods (1) expressively represent complicated dependencies, (2) forecast long horizons, and (3) efficiently train over long sequences. State-space models (SSMs) are classical models for time series, and prior works combine SSMs with deep learning layers for efficient sequence modeling. However, we find fundamental limitations with these prior approaches, proving their SSM representations cannot express autoregressive time series processes. We thus introduce SpaceTime, a new state-space time series architecture that improves all three criteria. For expressivity, we propose a new SSM parameterization based on the companion matrix – a canonical representation for discrete-time processes – which enables SpaceTime's SSM layers to learn desirable autoregressive processes. For long horizon forecasting, we introduce a "closed-loop" variation of the companion SSM, which enables SpaceTime to predict many future time-steps by generating its own layer-wise inputs. For efficient training and inference, we introduce an algorithm that reduces the memory and compute of a forward pass with the companion matrix. With sequence length ℓ and state-space size d, we go from Õ(d ℓ) naïvely to Õ(d + ℓ). In experiments, our contributions lead to state-of-the-art results on extensive and diverse benchmarks, with best or second-best AUROC on 6 / 7 ECG and speech time series classification, and best MSE on 14 / 16 Informer forecasting tasks. Furthermore, we find SpaceTime (1) fits AR(p) processes that prior deep SSMs fail on, (2) forecasts notably more accurately on longer horizons than prior state-of-the-art, and (3) speeds up training on real-world ETTh1 data by 73 Transformers and LSTMs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/11/2020

Stanza: A Nonlinear State Space Model for Probabilistic Inference in Non-Stationary Time Series

Time series with long-term structure arise in a variety of contexts and ...
research
09/26/2022

Liquid Structural State-Space Models

A proper parametrization of state transition matrices of linear state-sp...
research
05/31/2022

VQ-AR: Vector Quantized Autoregressive Probabilistic Time Series Forecasting

Time series models aim for accurate predictions of the future given the ...
research
10/26/2021

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

Recurrent neural networks (RNNs), temporal convolutions, and neural diff...
research
12/21/2017

On Long Memory Origins and Forecast Horizons

Most long memory forecasting studies assume that the memory is generated...
research
11/14/2022

Advancing the State-of-the-Art for ECG Analysis through Structured State Space Models

The field of deep-learning-based ECG analysis has been largely dominated...
research
03/13/2020

A Time Series Approach To Player Churn and Conversion in Videogames

Players of a free-to-play game are divided into three main groups: non-p...

Please sign up or login with your details

Forgot password? Click here to reset