Human motion forecasting has many useful applications. As intelligent systems should interact with humans, it becomes necessary to predict human movements and actions. Take, as one example, a cooperative task in which a human and a robot should safely collaborate in an assembly process. Therefore, multiple techniques have been developed to tackle such tasks. Ranging from simple heuristic approaches towards learning-based approaches. Machine learning approaches for such temporal data were mostly relying on recurrent networks that were applied to vectorized representations of joints. These were using skeletal structure only implicitly. The latest advances in geometric deep learning (GDL) provide a direct way to leverage the skeletal structure for predictions via graph convolutions (GCNs). In this work, we propose a straightforward implementation of such a model. We use spatio-temporal convolutions together with a type of GCN to extract spatio-temporal features in human motion, resulting in a simple autoregressive model. We adapt dilated causal convolutions for temporal modeling as used in[wavenet], but include local joint connectivity which leads to a lightweight spatio-temporal operation. The code will be made publicly available.111https://github.com/LucaHermes/lightweight-motion-forecasting
In this section, we introduce our deep geometric model for motion forecasting. The model is based on Graph-WaveNet [graph_wavenet], a spatio-temporal extension to the original WaveNet [wavenet].
Input to the model is a time series of consecutive skeletal states with joints and a -dimensional joint representation, (Fig. 1 shows the model architecture). In our experiments, we represent joints as quaternions, i.e. .
First, the linear input layer () is applied to each joint and acts as a trainable embedding for the -dimensional joint inputs. Secondly, consecutive Graph-WaveNet blocks extract spatio-temporal features from the given time series. Every block produces a skip output and the sum of these outputs are, thirdly, passed to an MLP that is applied per joint.
Every Graph-WaveNet block performs a spatio-temporal convolution (ST-Conv) followed by a purely spatial graph-convolution (K-GCN), as shown in Fig. 1. Both operations use the same output dimensionality, to which we refer to as the block dimensionality
. A residual connection is applied which bypasses both convolutions by adding the block input to the block output. The result is fed to the subsequent block.
We use the following model configuration in our experiments: The input layer consists of neurons. We use blocks with and a skip output dimensionality of . The two layers in the output MLP have and neurons, respectively. The resulting model has trainable parameters.
Spatio-Temporal Convolution In the original Graph-WaveNet a purely temporal convolution is used to extract temporal features from the input. In contrast, we substitute the temporal convolution that acts on each joint individually with a spatio-temporal convolution that integrates information from neighboring joints. This operation is inspired by the temporal extension module (TEM) [temporal_extension_module], which is a type of GCN placed prior to a temporal operation. Therefore, the temporal operation operates on information from the neighborhood .
We integrate the idea of TEM directly into the spatio-temporal convolution operation. This means, instead of applying a 1-dimensional convolution on the trajectory of a single joint, we apply a 2-dimensional spatio-temporal convolution on the trajectory of a kinematic chain of joints. The joint hierarchy is given by the kinematic tree of the skeleton. Through selecting the hip as the root joint, an ordering is introduced that extends into the leaf-joints, i.e. the head, hands and foot joints.
Fig. 1(b) visualizes our spatio-temporal convolution. First, the parent and grandparent joint for every joint are sampled based on the kinematic tree. Afterwards, the trajectories for these three joints are stacked and convolved using a convolutional kernel (blue rectangle). The kernel size is denoted as and in the spatial and temporal dimension, respectively. The applied convolution is causal and dilated in the temporal dimension following the pattern of [wavenet, graph_wavenet].
Spatial Graph-Convolution The purely spatial convolution is computed using graph convolutions (GCNs) as defined in [kipf_welling_17_spacial_gcn]. Instead of using the full skeletal graph, we again utilize the kinematic tree that was described above to convert the undirected skeletal graph into three directed subgraphs: In the first subgraph, an edge exists if and only if joint relative to joint is further up in the hierarchy of the kinematic tree. This subgraph retains all edges linking a joint to its immediate child joints. The second subgraph is similarly constructed, but with inverted edge direction, therefore retaining all edges linking a joint to its immediate parent joints. The third subgraph consists only of self-loops. The subgraphs are represented by three adjacency matrices . Note that
corresponds to the identity matrix. The output of this operation is computed as follows:
The inputs are given as , is the parameter matrix of the GCN operation and denotes the degree matrix of subgraph .
In this section, we describe the experiments we conducted on the presented model. To quantify the performance, we perform an evaluation using protocols from the related literature and provide qualitative results.
Dataset and Training Setup We train and evaluate the model on the Human3.6M (H3.6M) dataset [h36m_pami, h36m_ionescu]. The dataset consists of motion capture data from seven different human subjects performing 15 different actions. Table 1 and 2 provide an overview of these actions. We follow the general preprocessing procedure in [quaternet], downsampling the dataset by a factor of 2 to 25 Hz and mirror each trajectory across the --plane.
We split the data by subject into a train (subjects 1, 7, 8, 9, 11), validation (subject 6) and test set (subject 5). The models are trained using a batch size of 16 trajectories over 3000 epochs on the mean absolute error in the quaternion space. An epoch consists of five samples from every sequence. A sequence is a single trial of one subject, performing a single action. Each sample contains successive frames from a single downsampled or mirrored version of the trajectories. The model is trained using the Adam optimizer with an initial learning rate ofthat is decayed exponentially by a factor of after every epoch. We use a seed trajectory length of 32 frames () to condition the model, because this perfectly fits into the receptive field. The target sequence has a length of 10 frames () and was generated autoregressively.
Model Evaluation We show qualitative and quantitative results for the proposed model. For our model evaluation, we use the non-mirrored, but downsampled dataset. For the quantitative evaluation, we follow the standard evaluation protocol of [fragkiadaki2015]. This protocol first constructs an evaluation set that consists of four random samples from every sequence in the test set, i.e. the trials of subject 5. 222We were able to draw the same sequences as [fragkiadaki2015] using the RandomState implementation of the random package of NumPy (v. 1.19.2) with a seed of . An error metric quantifies the performance of the model on this evaluation set. The metric used by [fragkiadaki2015] is the Euclidean distance between the predicted and target rotations converted to Euler angles. The following equation summarizes this metric:
where is the number of time-steps, is the number of joints and the sum over accumulates the error in the , and dimension of the given Euler angles. The final results correspond to the average taken over the four samples. Following [martinez2017], the results for a running average over 2 and 4 frames (Run. avg. 2/4) and a zero-velocity-model are also documented as baselines. The zero-velocity model returns the first observed frame as the prediction for all successive frames.
Table 1 shows the results under the standard protocol for four actions. The running average and zero-velocity baselines are included, as well as results for multiple SoTA models. Table 2 lists additional results for the remaining 12 actions (we include results from other models when available for comparison). Our forecasting model generally shows competitive results that in some cases, e.g. eating, directions, and walkingtogether, outperform the referenced approaches. The original Graph-WaveNet architecture converges to a zero-velocity model.
|GRU sup. [martinez2017]||0.28||0.49||0.72||0.81||0.23||0.39||0.62||0.76||0.33||0.61||1.05||1.15||0.31||0.68||1.01||1.09|
|QuaterNet GRU abs./vel. [quaternet]||0.21||0.34||0.56||0.62||0.20||0.35||0.58||0.70||0.25||0.47||0.93||0.90||0.26||0.60||0.85||0.93|
|QuaterNet CNN abs./vel. [quaternet]||0.25||0.40||0.62||0.70||0.22||0.36||0.58||0.71||0.26||0.49||0.94||0.90||0.30||0.66||0.93||1.00|
|DCT-GCN short-term [dct_gcn]||0.18||0.31||0.49||0.56||0.16||0.29||0.50||0.62||0.22||0.41||0.86||0.80||0.20||0.51||0.77||0.85|
Fig. 3 visualizes predicted and true individual quaternion dimensions. Overall, the predictions are very similar to ground truth, but finer details are smoothed out. This is clearly visible in the trajectory of the right knee joint performing the walkingdog action. However, the model is still able to correctly predict the phase timing and thus model major movement features even in the long-term future ().
We combine the well-established causal dilated convolutions from WaveNet with geometric deep learning principles resulting in a lightweight autoregressive model. With about parameters, our model is an order of magnitude less complex compared to current models (cf. Table 1). Nonetheless, it shows competitive results compared to current approaches when evaluated on the H3.6M dataset for skeletal human motion forecasting. Furthermore, we show that this model is able to predict the phase timing even in a long-term forecasting setup.