State Aggregation Learning from Markov Transition Data

11/06/2018
by   Yaqi Duan, et al.
1

State aggregation is a model reduction method rooted in control theory and reinforcement learning. It reduces the complexity of engineering systems by mapping the system's states into a small number of meta-states. In this paper, we study the unsupervised estimation of unknown state aggregation structures based on Markov trajectories. We formulate the state aggregation of Markov processes into a nonnegative factorization model, where left and right factor matrices correspond to aggregation and disaggregation distributions respectively. By leveraging techniques developed in the context of topic modeling, we propose an efficient polynomial-time algorithm for computing the estimated state aggregation model. Under some "anchor state" assumption, we show that one can reliably recover the state aggregation structure from sample transitions with high probability. Sharp divergence error bounds are proved for the estimated aggregation and disaggregation distributions, and experiments with Manhattan traffic data are provided.

READ FULL TEXT
research
10/14/2018

Adaptive Low-Nonnegative-Rank Approximation for State Aggregation of Markov Chains

This paper develops a low-nonnegative-rank approximation method to ident...
research
02/08/2018

State Compression of Markov Processes via Empirical Low-Rank Estimation

Model reduction is a central problem in analyzing complex systems and hi...
research
07/12/2021

Polynomial Time Reinforcement Learning in Correlated FMDPs with Linear Value Functions

Many reinforcement learning (RL) environments in practice feature enormo...
research
04/12/2018

Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementations

In this paper we discuss policy iteration methods for approximate soluti...
research
02/13/2019

Sample-Optimal Parametric Q-Learning with Linear Transition Models

Consider a Markov decision process (MDP) that admits a set of state-acti...
research
06/01/2019

Learning low-dimensional state embeddings and metastable clusters from time series data

This paper studies how to find compact state embeddings from high-dimens...
research
06/16/2020

Iterative trajectory reweighting for estimation of equilibrium and non-equilibrium observables

We present two algorithms by which a set of short, unbiased trajectories...

Please sign up or login with your details

Forgot password? Click here to reset