1 Introduction
Discovering temporally hierarchical structure and representation in sequential data is the key to many problems in machine learning. In particular, for an intelligent agent exploring an environment, it is critical to learn such spatiotemporal structure hierarchically because it can, for instance, enable efficient optionlearning and jumpy future imagination, abilities critical to resolving the sample efficiency problem
(Hamrick, 2019). Without such temporal abstraction, imagination would easily become inefficient; imagine a person planning onehour driving from her office to home with future imagination at the scale of every second. It is also biologically evidenced that future imagination is the very fundamental function of the human brain (Mullally and Maguire, 2014; Buckner, 2010) which is believed to be implemented via hierarchical coding of the grid cells (Wei et al., 2015).There have been approaches to learn such hierarchical structure in sequences such as the HMRNN (Chung et al., 2016). However, as a deterministic model, it has the main limitation that it cannot capture the stochastic nature prevailing in the data. In particular, this is a critical limitation to imaginationaugmented agents because exploring various possible futures according to the uncertainty is what makes the imagination meaningful in many cases. There have been also many probabilistic sequence models that can deal with such stochastic nature in the sequential data (Chung et al., 2015; Krishnan et al., 2017; Fraccaro et al., 2017). However, unlike HMRNN, these models cannot automatically discover the temporal structure in the data.
In this paper, we propose the Hierarchical Recurrent State Space Model (HRSSM) that combines the advantages of both worlds: it can discover the latent temporal structure (e.g., subsequences) while also modeling its stochastic state transitions hierarchically. For its learning and inference, we introduce a variational approximate inference approach to deal with the intractability of the true posterior. We also propose to apply the HRSSM to implement efficient jumpy imagination for imaginationaugmented agents. We note that the proposed HRSSM is a generic generative sequence model that is not tied to the specific application to the imaginationaugmented agent but can be applied to any sequential data. In experiments, on 2D bouncing balls and 3D maze exploration, we show that the proposed model can model sequential data with interpretable temporal abstraction discovery. Then, we show that the model can be applied to improve the efficiency of imaginationaugmented agentlearning.
The main contributions of the paper are:
We propose the Hierarchical Recurrent State Space Model (HRSSM) that is the first stochastic sequence model that discovers the temporal abstraction structure.
We propose the application of HRSSM to imaginationaugmented agent so that it can perform efficient jumpy future imagination.
In experiments, we showcase the temporal structure discovery and the benefit of HRSSM for agent learning.
2 Proposed Model
2.1 Hierarchical Recurrent State Space Models
In our model, we assume that a sequence has a latent structure of temporal abstraction that can partition the sequence into nonoverlapping subsequences . A subsequence has length such that and . Unlike previous works (Serban et al., 2017), we treat the number of subsequences and the lengths of subsequences as discrete latent variables rather than given parameters. This makes our model discover the underlying temporal structure adaptively and stochastically.
We also assume that a subsequence is generated from a temporal abstraction and an observation has observation abstraction . The temporal abstraction and observation abstraction have a hierarchical structure in such a way that all observations in are governed by the temporal abstraction in addition to the local observation abstraction . As a temporal model, the two abstractions take temporal transitions. The transition of temporal abstraction occurs only at the subsequence scale while the observation transition is performed at every time step. This generative process can then be written as follows: p(X,S,L,Z,N) = p(N)iNp(X_i, S_i z_i, l_i) p(l_i  z_i) p(z_i  z_<i) where and
and the subsequence joint distribution
is: p(X_i, S_iz_i,l_i) = jl_ip(x_j^is_j^i)p(s_j^is_<j^i,z_i). We note that it is also possible to use the traditional Markovian state space model in Eqn. (2.1) and Eqn. (2.1) which has some desirable properties such as modularity and interpretability as well as having a closedform solution for a limited class of models like the linear Gaussian model. However, it is known that this Markovian model has difficulties in practice in capturing complex longterm dependencies (AugerMéthé et al., 2016). Thus, in our model, we take the recurrent state space model (RSSM) approach (Zheng et al., 2017; Buesing et al., 2018; Hafner et al., 2018b) which resolves this problem by adding a deterministic RNN path that can effectively encode the complex nonlinear longterm dependencies in the past, i.e., and in our model. Specifically, the transition is performed by the following updates: for , and for .2.2 Binary Subsequence Indicator
Although the above modeling intuitively explains the actual generation process, the discrete latent random variables
and —whose realization is an integer—raise difficulties in learning and inference. To alleviate this problem, we reformulate the model by replacing the integer latent variables by a sequence of binary random variables , called the boundary indicator. As the name implies, the role of this binary variable is to indicate whether a new subsequence should start at the next time step or not. In other words, it specifies the end of a subsequence. This is a similar operation to the FLUSH operation in the HMRNN model
(Chung et al., 2016). With the binary indicators, the generative process can be rewritten as follows: p(X,Z,S,M) = tTp(x_ts_t)p(m_ts_t)p(s_ts_, z_t, m_)p(z_tz_, m_) In this representation of the generative process, we can remove the subsequence hierarchy and make both transitions perform at every time step. Although this seemingly looks different to our original generation process, the control of the binary indicator—selecting either COPY or UPDATE—can make this equivalent to the original generation process, which we explain later in more detail. In Figure 1, we provide an illustration on how the binary indicators induce an equivalent structure represented by the discrete random variables
and .This reformulation has the following advantages. First, we do not need to treat the two different types of discrete random variables and separately but instead can unify them by using only one type of random variables . Second, we do not need to deal with the variable range of and because each time step has finite states while and depend on that can be changed across sequences. Lastly, the decision can be made adaptively while observing the progress of the subsequence, instead of making a decision governing the whole subsequence.
2.3 Prior on Temporal Structure
We model the binary indicator
by a Bernoulli distribution parameterized by
with a multilayer perceptron (MLP)
and a sigmoid function
. In addition, it is convenient to explicitly express our prior knowledge or constraint on the temporal structure using the boundary distribution. For instance, it is convenient to specify the maximum number of subsequences or the longest subsequence lengths when we do not want too many or too long subsequences. To implement, at each time step , we can compute the number of subsequences discovered so far by using a counter as well as the length of current subsequence with another counter . Based on this, we can design the boundary distribution with our prior knowledge as follows: p(m_t=1s_t)= {0 if n(m<t)≥Nmax,1 elseif l(m<t)≥lmax,σ(fmmlp(st))otherwise.2.4 Hierarchical Transitions
The transition of temporal abstraction should occur only a subsequence is completed. This timing is indicated by the boundary indicator. Specifically, the transition of temporal abstraction is implemented as follows: p(z_t  z_<t, m_<t)= {δ(zt=zt1)if mt1=0 (COPY),~p(ztct)otherwise (UPDATE) where is the following RNN encoding of all previous temporal abstractions (and ): c_t= {ct1if mt1=0 (COPY),fzrnn(zt1, ct1)otherwise (UPDATE). Specifically, having indicates that the time step is still in the same subsequence as the previous time step and thus the temporal abstraction should not be updated but copied. Otherwise, it indicates that the time step
was the end of the previous subsequence and thus the temporal abstraction should be updated. This transition is implemented as a Gaussian distribution
where both and are implemented with MLPs.At test time, we can use this transition of temporal abstraction without the COPY mode, i.e., every transition is UPDATE. This implements the jumpy future imagination which do not require to rollout at every raw time step and thus is computationally efficient.
The observation transition is similar to the transition of temporal abstraction except that we want to implement the fact that given the temporal abstraction , a subsequence is independent of other subsequences. The observation transition is implemented as follows: p(s_t  s_<t, z_t, m_<t) = ~p(s_t  h_t) where h_t= {fsrnn(st1∥ zt, ht1)if mt1=0 (UPDATE),fsmlp(zt)otherwise (INIT). Here, is computed by using an RNN to update (UPDATE), and a MLP to initialize (INIT). The concatenation is denoted by . Note that if the subsequence is finished, i.e., , we sample a new observational abstraction without conditioning on . That is, the underlying RNN is initialized.
3 Learning and Inference
As the true posterior is intractable, we apply variational approximation which gives the following evidence lower bound (ELBO) objective: logp(X)≥∑_M∫_Z,S q_ϕ(Z,S,MX) logpθ(X,Z,S,M)qϕ(Z,S,MX) This is optimized w.r.t. and using the reparameterization trick (Kingma and Welling, 2014). In particular, we use the Gumbelsoftmax (Jang et al., 2017; Maddison et al., 2017)
with straightthrough estimators
(Bengio et al., 2013) for the discrete variables . For the approximate posterior, we use the following factorization: q(Z,S,MX) = q(MX)q(ZM,X)q(SZ,M,X). That is, by sequence decomposition , we first infer the boundary indicators independent of and . Then, given the discovered boundary structure, we infer the two abstractions via the state inference and .Sequence Decomposition. Inferring the subsequence structure is important because the other state inference can be decomposed into independent subsequences. This sequence decomposition is implemented by the following decomposition: q(MX) = ∏_t=1^Tq(m_tX)=∏_t=1^TBern(m_tσ(φ(X))), where
is a convolutional neural network (CNN) applying convolutions over the temporal axis to extract dependencies between neighboring observations. This enables to sample all indicators
independently and simultaneously. Empirically, we found this CNNbased architecture working better than an RNNbased architecture.State Inference. State inference is also performed hierarchically. The temporal abstraction predictor does inference by encoding subsequences determined by and . To use the same temporal abstraction across the time steps of a subsequence, the distribution is also conditioned on the boundary indicator : q(z_t M, X)= {δ(zt=zt1)if mt1=0 (COPY),~q(zt ψfwdt1, ψbwdt)otherwise (UPDATE). We use the distribution to update the state . It is conditioned on all previous observations and this is represented by a feature extracted from a forward RNN . The other is a feature representing the current step’s subsequence that is extracted from a backward (masked) RNN . In particular, this RNN depends on , which is used as a masking variable, to ensure independence between subsequences.
The observation abstraction predictor is factorized and each observational abstraction is sampled from the distribution . The feature is extracted from a forward (masked) RNN that encodes the observation sequence and resets hidden states when a new subsequence starts.
4 Related Works
The most similar work with our model is the HMRNN (Chung et al., 2016). While it is similar in the sense that both models discover the hierarchical temporal structure, HMRNN is a deterministic model and thus has a severe limitation to use for an imagination module. In the switching statespace model (Ghahramani and Hinton, 2000)
, the upper layer is a Hidden Markov Model (HMM) and the behavior of the lower layer is modulated by the discrete state of the upper layer, and thus gives hierarchical temporal structure.
Linderman et al. (2016) proposed a new class of switching statespace models that discovers the dynamical units and also explains the switching behavior depending on observations or continuous latent states. The authors used inference based on messagepassing. The hidden semiMarkov models (Yu, 2010; Dai et al., 2016) perform similar segmentation with discrete states. However, unlike our model, there is no states for temporal abstraction. Kipf et al. (2018)proposed softsegmentation of sequence for compositional imitation learning.
The variational recurrent neural networks (VRNN)
(Chung et al., 2015) is a latent variable RNN but uses autoregressive state transition taking inputs from the observation. Thus, this can be computationally expensive to use as an imagination module. Also, the error can accumulate more severely in the high dimensional rollout. To resolve this problem, Krishnan et al. (2017) and Buesing et al. (2018) proposes to combine the traditional Markovian State Space Models with deep neural networks. Zheng et al. (2017) and Hafner et al. (2018a) proposed to use an RNN path to encode the past making nonMarkovian statespace models which can alleviate the limitation of the traditional SSMs. Serban et al. (2017) proposed a hierarchical version of VRNN called Variational Hierarchical Recurrent EncoderDecoder (VHRED) which results in a similar model as ours. However, it is a significant difference that our model learns the segment while VHRED uses a given structure. A closely related work is TDVAE (Gregor et al., 2019). TDVAE is trained on pairs of temporally separated time points. Jayaraman et al. (2019) and Neitz et al. (2018) proposed models that predict the future frames that, unlike our approach, have the lowest uncertainty. The resulting models predict a small number of easily predictable “bottleneck” frames through which any possible prediction must pass.5 Experiments
We demonstrate our model on visual sequence datasets to show (1) how sequence data is decomposed into perceptually plausible subsequences without any supervision, (2) how jumpy future prediction is done with temporal abstraction and (3) how this jumpy future prediction can improve the planning as an imagination module in a navigation problem. Moreover, we test conditional generation where is the context observation of length . With the context, we preset the state transition of the temporal abstraction by deterministically initializing with implemented by a forward RNN.
5.1 Bouncing Balls
We generated a synthetic 2D visual sequence dataset called bouncing balls
. The dataset is composed of two colored balls that are designed to bounce in hitting the walls of a square box. Each ball is independently characterized with certain rules: (1) The color of each ball is randomly changed when it hits a wall and (2) the velocity (2D vector) is also slightly changed at every time steps with a small amount of noise. We trained a model to learn 1D state representations and all observation data
are encoded and decoded by convolutional neural networks. During training, the length of observation sequence data is set to and the context length is . Hyperparameters related to sequence decomposition are set as and .Our results in Figure 5 show that the sequence decomposer predicts reasonable subsequences by setting a new subsequence when the color of balls is changed or the ball is bounced. At the beginning of training, the sequence decomposer is unstable with having large entropy and tends to define subsequences with a small number of frames. It then began to learn to increase the length of subsequences and this is controlled by annealing the temperature of Gumbelsoftmax towards small values from 1.0 to 0.1. However, without our proposed prior on temporal structure, the sequence decomposer fails to properly decompose sequences and our proposed model consequently converges into RSSM.
5.2 Navigation in 3D Maze
Another sequence dataset is generated from the 3D maze environment by an agent that navigates the maze. Each observation data is defined as a partially observed view observed by the agent. The maze consists of hallways with colored walls and is defined on a grid map as shown in Figure 7. The agent is set to navigate around this environment and the viewpoint of the agent is constantly jittered with some noise. We set some constraints on the agent’s action (forward, leftturn, rightturn) that the agent is not allowed to turn around when it is located on the hallway. However, it can turn around when it arrives nearby intersections between hallways. Due to these constraints, the agent without a policy can randomly navigate the maze environment and collect meaningful data.
To train an environment model, we collected 1M steps (frames) from the randomly navigating agent and used it to train both RSSM and our proposed HRSSM. For HRSSM, we used the same training setting as bouncing balls but different and for the sequence decomposition. The corresponding learning curves are shown in Figure 6 that both reached a similar ELBO. This suggests that our model does not lose the reconstruction performance while discovering the hierarchical structure. We trained state transitions to be actionconditioned and therefore this allows to perform actioncontrolled imagination. For HRSSM, only the temporal abstraction state transition is actionconditioned as we aim to execute the imagination only with the jumpy future prediction. The overall sequence generation procedure is described in Figure 7. The temporal structure of the generated sequence shows how the jumpy future prediction works and where the transitions of temporal abstraction occur. We see that our model learns to set each hallway as a subsequence and consequently to perform jumpy transitions between hallways without repeating or skipping a hallway. In Figure 8, a set of jumpy predicted sequences from the same context and different input actions are shown and this can be interpreted as imaginations the agent can use for planning.
GoalOriented Navigation
We further use the trained model as an imagination module by augmenting it to an agent to perform the goaloriented navigation. In this experiment, the task is to navigate to a randomly selected goal position within the given life steps. The goal position in the grid map is not provided to the agent, but a
cropped image around the goal position is given. To reach the goal fast, the agent is augmented with the imagination model and allowed to execute a rollout over a number of imagination trajectories (i.e., a sequence of temporal abstractions) by varying the input actions. Afterward, it decides the best trajectory that helps to reach the goal faster. To find the best trajectory, we use a simple strategy: a cosinesimilarity based matching between all imagined state representations in imaginations and the feature of the goal image. The feature extractor for the goal image is jointly trained with the model.
^{1}^{1}1During training, the window (image) around the agent position is always given as additional observation data and we trained feature extractor by maximizing the cosinesimilarity between the extracted feature and the corresponding time step state representation. This way, at every time step we let the agent choose the first action resulting in the best trajectory. This approach can be considered as a simple variant of the Monte Carlo Tree Search (MCTS) and the detailed overall procedure can be found in Appendix. Each episode is defined by randomly initializing the agent position and the goal position. The agent is allowed maximum 100 steps to reach the goal and the final reward is defined as the number of remaining steps when the agent reaches the goal or consumes all lifesteps. The performance highly depends on the accuracy and the computationally efficiency of the model and we therefore compare between RSSM and HRSSM with varying the length of imagined trajectories. We measure the performance by randomly generated 5000 episodes and show how each setting performs across the episodes by plotting the reward distribution in Figure 10. It is shown that the HRSSM significantly improves the performance compared to the RSSM by having the same computational budget.HRSSM showed consistent performance over different lengths of imagined trajectories and most episodes were solved within 50 steps. We believe that this is because HRSSM is able to abstract multiple time steps within a single state transition and this enables to reduce the computational cost for imaginations. The results also show that finding the best trajectory becomes difficult as the imagination length gets larger, i.e., the number of possible imagination trajectories increases. This suggests that imaginations with temporal abstraction can benefit both the accuracy and the computationally efficiency in effective ways.
6 Conclusion
In this paper, we introduce the Variational Temporal Abstraction (VTA), a generic generative temporal model that can discover the hierarchical temporal structure and its stochastic hierarchical state transitions. We also propose to use this temporal abstraction for temporallyextended future imagination in imaginationaugmented agentlearning. Experiment results shows that in general sequential data modeling, the proposed model discovers plausible latent temporal structures and perform hierarchical stochastic state transitions. Also, in connection to the modelbased imaginationaugmented agent for a 3D navigation task, we demonstrate the potential of the proposed model in improving the efficiency of agentlearning.
Acknowledgments
We would like to acknowledge Kakao Brain cloud team for providing computing resources used in this work. TK would also like to thank colleagues at Mila, Kakao Brain, and Rutgers Machine Learning Group. SA is grateful to Kakao Brain, the Center for Super Intelligence (CSI), and Element AI for their support. Mila (TK and YB) would also like to thank NSERC, CIFAR, Google, Samsung, Nuance, IBM, Canada Research Chairs, Canada Graduate Scholarship Program, and Compute Canada.
References
 Statespace models’ dirty little secrets: even simple linear gaussian models can have estimation problems. Scientific reports 6, pp. 26677. Cited by: §2.1.

Estimating or propagating gradients through stochastic neurons for conditional computation
. CoRR abs/1308.3432. External Links: Link, 1308.3432 Cited by: §3.  The role of the hippocampus in prediction and imagination. Annual review of psychology 61, pp. 27–48. Cited by: §1.

Learning and querying fast generative models for reinforcement learning
. arXiv preprint arXiv:1802.03006. Cited by: §2.1, §4. 
Learning phrase representations using rnn encoder–decoder for statistical machine translation.
In
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)
, Doha, Qatar, pp. 1724–1734. External Links: Link Cited by: Appendix C.  Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01704. Cited by: §1, §2.2, §4.
 A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980–2988. Cited by: §1, §4.
 Fast and accurate deep network learning by exponential linear units (elus). See DBLP:conf/iclr/2016, External Links: Link Cited by: Appendix C.
 Recurrent hidden semimarkov model. Cited by: §4.

A disentangled recognition and nonlinear dynamics model for unsupervised learning
. In Advances in Neural Information Processing Systems, pp. 3601–3610. Cited by: §1.  Variational learning for switching statespace models. Neural computation 12 (4), pp. 831–864. Cited by: §4.
 Temporal difference variational autoencoder. In International Conference on Learning Representations, Cited by: §4.
 Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551. Cited by: §4.
 Learning latent dynamics for planning from pixels. CoRR abs/1811.04551. External Links: 1811.04551 Cited by: §2.1.

Analogues of mental simulation and imagination in deep learning
. Current Opinion in Behavioral Sciences 29, pp. 8 – 16. Note:SI: 29: Artificial Intelligence (2019)
External Links: ISSN 23521546 Cited by: §1.  Categorical reparameterization with gumbelsoftmax. In International Conference on Learning Representations, Cited by: §3.
 Timeagnostic prediction: predicting predictable video frames. In International Conference on Learning Representations, External Links: Link Cited by: §4.
 Autoencoding variational bayes. See DBLP:conf/iclr/2014, Cited by: §3.
 Glow: generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett (Eds.), pp. 10215–10224. External Links: Link Cited by: Appendix C.
 Compositional imitation learning: explaining and executing one task at a time. CoRR abs/1812.01483. External Links: Link, 1812.01483 Cited by: §4.
 Structured inference networks for nonlinear state space models. In ThirtyFirst AAAI Conference on Artificial Intelligence, Cited by: §1, §4.
 Recurrent switching linear dynamical systems. arXiv preprint arXiv:1610.08466. Cited by: §4.
 The concrete distribution: a continuous relaxation of discrete random variables. In International Conference on Learning Representations, Cited by: §3.
 Memory, imagination, and predicting the future: a common brain mechanism?. The Neuroscientist 20 (3), pp. 220–234. Cited by: §1.
 Adaptive skip intervals: temporal abstraction for recurrent dynamical models. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett (Eds.), pp. 9816–9826. Cited by: §4.
 On the convergence of adam and beyond. In International Conference on Learning Representations, External Links: Link Cited by: Appendix C.
 A hierarchical latent variable encoderdecoder model for generating dialogues. In ThirtyFirst AAAI Conference on Artificial Intelligence, Cited by: §2.1, §4.
 A principle of economy predicts the functional architecture of grid cells. Elife 4, pp. e08362. Cited by: §1.
 Hidden semimarkov models. Artif. Intell. 174 (2), pp. 215–243. External Links: ISSN 00043702, Document Cited by: §4.
 State space LSTM models with particle MCMC inference. arXiv preprint arXiv:1711.11179. Cited by: §2.1, §4.
Appendix A ActionConditioned Temporal Abstraction State Transition
We implement actionconditioned state transition p(z_t  a_t, z_<t, m_<t)= {δ(zt=zt1)if mt1=0 (COPY),~p(ztct)otherwise (UPDATE) where the action input is only affecting the UPDATE operation and we feed it into the deterministic path as the following: c_t= {ct1if mt1=0 (COPY),fz(zt1∥ at, ct1)otherwise (UPDATE).
Appendix B GoalOriented Navigation
Appendix C Implementation Details
For bouncing balls, we define the reconstruction loss (data likelihood) by using binary crossentropy. The images from 3D maze are preprocessed by reducing the bit depth to 5 bits (Kingma and Dhariwal, 2018) and therefore the reconstruction loss is computed by using Gaussian distribution. We used the AMSGrad (Reddi et al., 2018), a variant of Adam, optimizer with learning rate and all minibatchs are with 64 sequences with length . Both CNNbased encoder and decoder are composed of 4 convolution layers with ELU activations (Clevert et al., 2016). A GRU (Cho et al., 2014) is used for all RNNs with 128 hidden units. The state representations of temporal abstraction and observation abstraction are sampled from 8dimensional diagional Gaussian distributions.
Appendix D Evidence Lower Bound (ELBO)
We derive the ELBO without considering recurrent deterministic paths.
Comments
There are no comments yet.