Human action recognition has many useful applications such as human computer interaction, smart video surveillance, sports and health care. These applications are one of motivations behind much research work devoted to this area in the last few years. However, even with a large number of research papers found in the literature, many challenging problems, such as different viewpoints, visual appearances, human body sizes, lighting conditions, and speeds of action execution, still affect the performance of these algorithms. Further problems include partial occlusion of the human subjects by other objects in the scene and self-occlusion of human subjects themselves. Among the human action recognition papers presented in the literature, some of the early techniques focus on using conventional RGB videos [1, 2, 3, 4, 5, 6, 7, 8]. While these video-based techniques gave promising results, their recognition accuracy is still relatively low, even when the scene is free of clutter.
The Kinect camera introduced by Microsoft in 2001 was an attempt to broaden the 3D gaming experience of the Xbox 360’s audience. However, as the Kinect camera can capture real-time RGB and depth videos, and there is a publicly available toolkit for computing the human skeleton model from each frame of a depth video, many research papers on 3D human action recognition using the Kinect camera have emerged. One advantage of using depth videos than the conventional RGB videos is that it is easier to segment the foreground human subject even when the scene is cluttered. As depth videos do not have colour information, the colour of the clothes worn by the human subject has no effect on the segmentation process. This allows action recognition researchers to focus their effort more on getting robust feature descriptors to describe the actions rather than on low level segmentation. Numerous representative methods for 3D action analysis using depth videos include [9, 10, 11, 12, 13, 14]
. These methods employ advanced machine learning techniques for which good results have been reported. Of course, depth images are also vulnerable to noise due to various factors. Thus, using depth images does not always guarantee good action recognition performance . The algorithm used for computing the 3D joint positions of the human skeletal model by the Kinect toolkit is based on the human skeleton tracking framework (OpenNI) of Shotton et al. . In addition to the availability of the real-time depth video stream, this tracking framework also opens up the research area of skeleton-based human action recognition [17, 18, 19, 20, 21, 22].
Human action recognition methods using the Kinect data can be classified into two categories, based on how the feature descriptors are extracted to represent the human actions. The first category is handcrafted features. Action recognition methods using handcrafted features require two complex hand-design stages, namelyfeature extraction and feature representation
, to build the final descriptor. Both the feature extraction stage and feature representation stage differ from one method to another. The feature extraction stage may involve computing the depth (and/or colour) gradients, histogram, and other more complex transformations of the video data. The feature representation stage may involve simple concatenation of the feature components extracted from the previous stage, a more complex fusion step of these feature components, or even using a machine learning technique, to get the final feature descriptor. These methods usually involve a number of operations that require researchers to carry out careful feature engineering and tuning. Kinect-based human action recognition algorithms using handcrafted features reported in the literature include[9, 16, 10, 11, 12, 13, 17, 19, 23, 24].
The second category is deep learning features. With the huge advance in neural network research in the last decade, deep neural networks have been used to extract high-level features from video sequences for many different applications, including 3D human action analysis. Deep learning methods reduce the need for feature engineering; however, they require a huge amount of labelled training data, which may not be available, and a long time to train. For small human action recognition datasets, deep learning methods may not give the best performance. Recent Kinect-based human action recognition algorithms are:[18, 14, 22, 25, 20, 21, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38].
Research contributions. Although both handcrafted and deep learning features have been used in human action recognition, to the best of our knowledge, a thorough comparison of recent action recognition methods for these two categories is not found in the literature. Our contributions in this paper are twofold:
We evaluate the performance of 10 recent state-of-art human action recognition algorithms, with specific focus on comparing the effectiveness of using handcrafted features versus deep learning features and skeleton-based features versus depth-based features. We believe that there is a lack of such a comparison in the literature on human action recognition.
Furthermore, we evaluate the cross-view versus cross-subject performance of these algorithms and, for the multiview datasets, the impact of the camera view for both small and large datasets on human action recognition with respect to whether the features being used are depth-based, skeleton-based, or depth+skeleton-based. To the best of our knowledge, such evaluation has not been performed before.
The paper is organized as follows. Section II gives a brief review on recent human action recognition techniques. Section III covers the details of the 10 algorithms being compared in this paper. In Section IV, we describe our experimental setting and the benchmark datasets. Sections V and VI summarize our experimental results, comparison, and discussions. The last section concludes the paper.
Ii Related Work
|Algorithms||Year||Short descriptions||Kinect data used||Feature dimension|
|HON4D (Oreifej & Liu) ||CVPR 2013||handcrafted (global descriptor)||depth|
|HDG (Rahmani et al.) ||WACV 2014||handcrafted (local + global descriptor)||depth+skeleton|
|LARP-SO (Vemulapalli & Chellappa) ||CVPR 2016||handcrafted (Lie Group)||skeleton||#frames|
|HOPC (Rahmani et al.) ||TPAMI 2016||handcrafted (local descriptor)||depth pointcloud||depending on #STKs|
|SCK+DCK (Koniusz et al.) ||ECCV 2016||
handcrafted (tensor representations)
|P-LSTM (Shahroudy et al.) ||CVPR 2016||deep learning (LSTM)||skeleton||#joints|
|HPM+TM (Rahmani & Mian) ||CVPR 2016||deep learning (CNN)||depth||4096|
|Clips+CNN+MTLN (Ke et al.) ||CVPR 2017||deep learning (pre-trained VGG19, MTLN)||skeleton||7168|
|IndRNN (Li et al.) ||CVPR 2018||deep learning (independently RNN)||skeleton||512|
|ST-GCN (Yan et al.) ||AAAI 2018||deep learning (Graph ConvNet)||skeleton||256|
STK stands for spatio-temporal keypoint. The P-LSTM features include 8 video segments, each of which is composed of a number of 3D joints.
Action recognition methods can be classified into three categories based on the type of input data: colour-based [1, 3, 2, 39, 40, 4, 6, 5, 8, 7, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55], depth-based [9, 10, 56, 11, 57, 12, 58, 59, 60, 61, 25, 62, 63, 13, 14, 47, 64, 22, 55, 65, 24], and skeleton-based [16, 66, 17, 67, 68, 69, 70, 23, 19, 71, 18, 22, 21, 20, 55, 29, 35, 34, 33, 32, 31, 30, 28, 36, 37, 38, 72, 24, 73, 74, 75, 76]. In this section, we will focus on reviewing recent methods using the last two types of features.
Depth-based action recognition. Action recognition from depth videos [9, 10, 56, 11, 57, 60, 58, 59, 13, 63] has become more popular because of the availability of real-time cost-effective sensors. Most existing depth-based action recognition methods use global features such as space-time volume and silhouette information. For example, Oreifej and Liu  captured the discriminative features by projecting the 4D surface normals obtained from the depth sequence onto a 4D regular space to build the Histogram of Oriented 4D Normals (HON4D). Yang and Tian  extended HON4D by concatenating local neighbouring hypersurface normals from the depth video to jointly characterize local shape and motion information. More precisely, they introduced an adaptive spatio-temporal pyramid to subdivide the depth video into a set of space-time cells for more discriminative features. Xia and Aggarwal  proposed to filter out the noise from the depth sensor so as to get more reliable spatio-temporal interest points for action recognition. Although these methods have achieved impressive performance for frontal action recognition, they are sensitive to changes of viewpoint. One way to alleviate this viewpoint issue is to directly process the pointclouds, as reported in the paper by Rahmani et al. .
Apart from the methods mentioned above which use handcrafted features, the use of deep learning features [61, 25, 62, 47, 14, 22, 64, 65] in human action recognition is on the rise. For example, Wang et al.  used a Hierarchical Depth Motion Maps (HDMMs) to extract the body shape and motion information and then trained a 3-channel deep Convolutional Neural Network (CNN) on the HDMMs for human action recognition. In the following years, Rahmani and Mian  proposed to train a single Human Pose Model (HPM) from real motion capture data to transfer the human pose from different unknown views to a view-invariant feature space, and Zhang et al.  used a multi-stream deep neural networks to jointly learn the semantic relations among action attributes.
Skeleton-based action recognition. Existing skeleton-based action recognition methods can be grouped into two categories: joint-based methods and body part based methods. Joint-based methods model the positions and motion of the joints (either individual or a combination) using the coordinates of the joints extracted by the OpenNI tracking framework. For instance, a reference joint may be used and the coordinates of other joints are defined relative to the reference joint [10, 17, 21, 20], or the joint orientations may be computed relative to a fixed coordinate system and used to represent the human pose , etc. For the body part based methods, the human body parts are used to model the human’s articulated system. These body parts are usually modelled as rigid cylinders connected by joints. Information such as joint angles , temporal evolution of body parts [71, 18, 23, 26], and 3D relative geometric relationships between rigid body parts [17, 19, 26] has all been used to represent the human pose for action recognition.
The method proposed by Vemulapalli et al.  falls into the body part based category. They represent the relative geometry between a pair of body parts, which may or may not be directly connected by a joint, as a point in . Thus, a human skeleton is a point of the Lie group where each action corresponds to a unique evolution of such a point in time. The approach of Ke et al. 
relies on both body parts and body joints. The human skeleton model was divided into 5 body parts. A specific joint was selected for each body part as the reference joint and the coordinates of other joints were expressed as vectors relative to that reference joint. Various distance measures were computed from these vectors to yield a feature vector for each video frame. The features vectors from all video frames were finally appended together and scaled to form a handcrafted greyscale image descriptor fed into a CNN. Somewhat related approach uses kernels formed over body joints to obtain feature maps fed into a CNN for simultaneous action recognition and domain adaptation.
Recent human action recognition papers favour deep learning techniques to perform human action recognition. Apart from the CNN-based approaches [20, 77], Recurrent Neural Networks (RNNs) have also been popular [67, 68, 18, 69, 70, 27, 65, 35, 38]. Since Long Short-term Memory (LSTM)  can model temporal dependencies as RNNs and even capture the co-occurrences of human joints, LSTM networks have also been a popular choice in human action recognition [31, 33, 37, 36, 73]. For instance, Zhu et al.  presented an end-to-end deep LSTM network with a dropout step, Shahroudy et al.  proposed a Part-aware Long Short-term Memory (P-LSTM) network to learn the long-term patterns of the 3D trajectories for each grouped body part, and Liu et al.  introduced the use of trust gates in their spatio-temporal LSTM architecture.
Action recognition via a combination of skeleton and depth features. Combining skeleton and depth features together helps overcome situations when there are interactions between human subject and other objects or when the actions have very similar motion trajectories. Various action recognition algorithms [12, 47, 22, 72] that use both depth and skeleton features for robust human action recognition have been reported in recent years. For example, Rahmani et al.  proposed to combine 4 types of local features extracted from both depth images and 3D joint positions to deal with local occlusions and to increase the recognition accuracy. We refer to their method as HDG from hereon. Another example is the approach of Shahroudy et al.  where Local Occupancy Patterns (LOP), HON4D, and skeleton-based features are combined with hierarchical mixed norms which regularize the weights in each modality group of each body part. Recently, Rahmani and Bennamoun  used an end-to-end deep learning model to learn the body part representation from both skeletal and depth images. To improve the performance of the model, they adopted a bilinear compact pooling  layer for the generated depth and skeletal features. Elmadany et al. , on the other hand, proposed to use canonical correlation analysis to maximize the correlation of features extracted from different sensors. The features investigated in their paper include bag of angles extracted from skeleton data, depth motion map from depth video, and optical flow from RGB video. The subspace shared by all the features was learned and average pooling was used to get the final feature descriptor.
Iii Analyzed and Evaluated Algorithms
We chose ten action recognition algorithms shown in Table I for our comparison and evaluation as they are recent action recognition methods (from 2013 onward) and they use skeleton-based, depth-based, handcrafted, and/or deep learning features. The technical details of these algorithms are summarized below.
HON4D. Oreifej and Liu  presented a global feature descriptor that captures the geometry and motion of human action in the 4D space of spatial coordinates, depth and time. To form the HON4D descriptor, the space was quantized using a 600-cell polychoron with 120 vertices. The vectors stretching from the origin to these vertices were used as projection axes to obtain the distribution of normals for each video sequence. To improve the classification performance, random perturbations were added to those projectors. The dimensions of HON4D features (Table I) vary across different datasets.
HDG. In this algorithm , each depth sequence was firstly divided into small subvolumes; the histograms of depth and depth derivatives were computed for each subvolume. For each skeleton sequence, the torso joint was used as a stable reference joint for computing the histograms of joint position differences. In addition, the variations of each joint movement volume were incorporated into the global feature vector to form spatio-temporal joint features. Two Random Decision Forests (RDFs) were trained in this algorithm, one for feature pruning and one for classification. More details about feature dimensions of HDG and the feature pruning applied by us will be given in Section IV-D.
HOPC. Approach  models depth images as 3D pointclouds. The authors used two types of support volume, namely, so-called spatial support volume and spatio-temporal support volume. The HOPC descriptor was extracted from the pointcloud falling inside the support volume around each point, which may be classified as a spatio-temporal Keypoint
(STK) if the eigenvalue ratios of the pointcloud around it are larger than some predefined threshold. For each STK, the algorithm further projected eigenvectors onto the axes of the 20 vertices of a regular dodecahedron. The final HOPC descriptor for each STK is a concatenation of 3 small histograms, each of which captures the distribution of an eigenvector of the pointcloud within the support volume.
LARP-SO. Vemulapalli and Chellappa  extended their earlier work  by the use of Lie Algebra Relative Pairs via for action recognition. We follow the convention adopted in  and name this algorithm as LARP-SO. In this algorithm, the rolling map, which describes how a Riemannian manifold rolls over another one along a smooth rolling curve, was used for 3D action recognition. Each skeleton sequence was firstly represented by the relative 3D rotations between various human body parts, and each action was then modelled as a curve in the Lie Group. Since it is difficult to perform the classification of action curves in a non-Euclidean space, the curves were unwrapped by the logarithm map at a single point while a rolling map was used to reduce distortions. The Fourier Temporal Pyramid (FTP) representation  was used in the algorithm to make the descriptor more robust to noise and less sensitive to temporal misalignments.
SCK+DCK. Koniusz et al.  used tensor representations to capture the higher-order relationships between 3D human body joints for action recognition. They applied two different RBF kernels which they referred to as Sequence Compatibility Kernel (SCK) and Dynamics Compatibility Kernel (DCK). The former kernel captures the spatio-temporal compatibility of joints while the latter models the action dynamics of a sequence. An SVM was then trained on linearized feature maps of such kernels for action classification.
HPM+TM. Approach  employs a dictionary containing representative human poses from a motion capture database. A deep CNN architecture which is a modification of  was then used to train a view-invariant human pose model. Real depth sequences were passed to the learned model frame-by-frame to extract high-level view-invariant features. Similarly to the LARP-SO algorithm above, the FTP was used to capture the temporal structure of the action videos. The final descriptor for each video sequence is a collection of the Fourier coefficients from all the segments.
P-LSTM. Approach  proceeds by transforming 3D coordinates of the body joints from the camera coordinate system to the body coordinate system with the origin set at the spine. The 3D coordinates of all other body joints were then scaled based on the distance between the ‘hip centre’ joint and the ‘spine’ joint. A P-LSTM model was built by splitting the memory cells from the LSTM model into body part based sub-cells. For each video sequence, the pre-processed human joints were grouped into 5 parts (torso, two hands, and two legs) and the video was divided into 8 equal-sized video segments. Then, for a randomly selected frame per video segment, 3D coordinates of the joints inside each grouped part were concatenated and passed as input to the P-LSTM network to learn common temporal patterns of the parts and combine them into a global representation.
Clips+CNN+MTLN. Ke et al.  presented a skeletal representation referred to as clips. The method proceeds by transforming the Cartesian coordinates of human joints (per skeleton sequence) into the cylindrical coordinates to generate 3 clips, with each clip corresponding to one channel of the cylindrical coordinates. To encode the temporal information for the whole video sequence, four stable joints (left shoulder, right shoulder, left hip and right hip) were selected as reference joints to produce 4 coordinate frames. The pre-trained VGG19 network  was used as a feature extractor to learn the long-term spatio-temporal features from intermediate images formed from the 4 coordinate frames. Moreover, approach  also employs the Multi-task Learning Network (MTLN) proposed by  to incorporate the spatial structural information from the CNN features.
IndRNN. Li et al. 
proposed a new RNN method, an Independently Recurrent Neural Network, for which neurons per layer are independent of each other but they are reused across layers. Finally, multiple IndRNNs were stacked to build a deeper network than the traditional RNN.
ST-GCN. The spatio-temporal graph representation for skeleton sequences proposed by Yan et al.  is an extension of Graph Convolutional Networks (GCN) [82, 83, 84] tailored to perform human action recognition. Firstly, the spatio-temporal graph is constructed by inserting edges between neighbouring body joints (nodes) of the human body skeleton as well as along the temporal direction. Subsequently, GCN and a classifier are applied to infer dependencies in the graphs (a single graph corresponds to a single action sequence) and perform classification.
Iv Experimental Setting
To perform experiments, we obtained off-the-shelf codes for HON4D , HOPC , LARP-SO , HPM+TM , IndRNN  and ST-GCN  from the respective authors’ websites. For SCK+DCK , HDG , P-LSTM  and Clips+CNN+MTLN , we used our own Matlab implementations given that codes for these methods are not publicly available. Moreover, we employed ten variants of the HDG  representation so as to evaluate the performance with respect to different combinations of its individual descriptor types. We also implemented the traditional RNN and LSTM as baseline methods, and added four variants of P-LSTM to evaluate the impact of using different numbers of video segments for skeletal representation as well as different numbers of hidden neurons.
Iv-a Benchmark Datasets
Listed in Table II are six benchmark datasets used in our evaluation, each of which is detailed below.
|MSRAction3D ||2010||20||10||1||567||Kinect v1||Depth + 3DJoints||20|
|3D Action Pairs ||2013||12||10||1||360||Kinect v1||RGB + Depth + 3DJoints||20|
|CAD-60 ||2011||14||4||–||68||Kinect v1||RGB + Depth + 3DJoints||15|
|UWA3D Activity Dataset ||2014||30||10||1||701||Kinect v1||RGB + Depth + 3DJoints||15|
|UWA3D Multiview Activity II ||2015||30||9||4||1070||Kinect v1||RGB + Depth + 3DJoints||15|
|NTU RGB+D Dataset ||2016||60||40||80||56880||Kinect v2||RGB + Depth + IR + 3DJoints||25|
(The number of views is not stated in the CAD-60 dataset.)
MSRAction3D  is one of the earliest action datasets captured with the Kinect depth camera. It contains 20 human sport-related activities such as jogging, golf swing and side boxing. Each action in this dataset was performed 2 or 3 times by 10 people. This dataset is challenging because of high inter-action similarities.
3D Action Pairs  contains 6 selected pairs of actions that have very similar motion trajectories, e.g., put on a hat and take off a hat; pick up a box and put down a box; stick a poster and remove a poster. Each action was performed 3 times by 10 people. There are two challenging aspects of this dataset: (i) the actions in each pair have similar motion trajectories; (ii) the object that is interacted by the subject in each video is only present in the RGB-D data but not the skeleton data.
Cornell Activity Dataset (CAD)  comprises two sub-datasets, CAD-60 and CAD-120. Both sub-datasets contain RGB-D and tracked skeleton video sequences of human activities captured by a Kinect sensor. In this paper, only CAD-60 was used in the experiments. Fig. 1 illustrates depth images from the CAD-60 dataset and demonstrates that this dataset exhibits high levels of noise in its depth videos.
UWA3D Activity Dataset  contains 30 actions performed by 10 people of various height at different speeds in cluttered scenes. This dataset has high between-class similarity and contains frequent self-occlusions.
UWA3D Multiview Activity II  contains 30 actions performed by 9 people in a cluttered environment. In this dataset, the Kinect camera was moved to different positions to capture the actions from 4 different views (see Fig. 2(a)-(c)): front view (), left view (), right view (), and top view (). This dataset is therefore more challenging than the previous four datasets. Fig. 2(d) shows sample video frames from this dataset.
NTU RGB+D Dataset  is so far the largest Kinect-based action dataset which contains 56,880 video sequences and over 4 million frames. There are 60 action classes performed by 40 subjects captured from 80 views with 3 Kinect v.2 cameras. This dataset has variable sequence lengths for different sequences and exhibits high intra-class variations.
Dataset usage. Below we detail how the above six datasets were used in our experiments.
The MSRAction3D, 3D Action Pairs, CAD-60 and UWA3D Activity datasets were used for cross-subject (single-view) experiments. For every dataset, we used half of the subjects’ data for training and the remaining half for testing. We tested all the possible combinations of subjects for the training and testing splits to obtain the average recognition accuracy of each algorithm. For example, for 10 subjects in the MSRAction3D dataset, experiments were carried out.
The UWA3D Multiview Activity II dataset was used for cross-view experiments, with two views of the samples being used for training and the remaining views for testing. There were 12 different view combinations in the experiments.
The NTU RGB+D dataset was used in both cross-subject and cross-view experiments. Despite indications that this dataset has 80 views of human action recognition, the data samples were grouped according to three camera sets. For cross-view action recognition, we used the video sequences captured by two cameras as our training data and the remaining sequences for testing. A total of 3 different camera combinations were experimented with.
Iv-B Evaluation Settings
Below we detail the experimental settings of the algorithms.
HON4D. According to , HON4D has the frame size of and each video is divided into () spatio-temporal cells. In our evaluations, we used these same settings for all the datasets.
HOPC. Rahmani et al.  used different spatial and temporal scales for different datasets. In this paper, a constant temporal scale and spatial scale were used for all the datasets. For the MSRAction3D and 3D Action Pairs datasets, the temporal and spatial scales were set to 2 and 19, respectively. For the remaining datasets, we used 2 for the temporal scale and 140 as the spatial scale. Moreover, we divided each depth video into spatio-temporal cells (along the , and time axes) to extract features.
LARP-SO. The desired number of frames  used for computing skeletal representation varies depending on the datasets used in the experiments. The desired frame numbers for the UWA3D Activity, UWA3D Multiview Activity II, and NTU RGB+D datasets were all set to 100. For the MSRAction3D, 3D Action Pairs datasets, and CAD-60, they were set to 76, 111, and 1,000, respectively.
SCK+DCK. We followed the experimental settings described in [23, 85] for all the datasets and we used authors’ newest model which aggregates over subsequences (not just sequences). For SCK, we normalized all human body joints with respect to the hip joints across frames as well as the lengths of all body parts. For DCK, we used the unnormalized body joints, and assumed that the displacements of body joint coordinates across frames captured their temporal evolution.
HDG. According to ), the number of used subvolumes has no significant effect to the discriminative features, we divided each video sequence into subvolumes (along , and time) for computing the histograms of depth as well as the depth gradients. For the joint movement volume features, each joint volume was divided into cells (along , and time). There are four individual feature representations encapsulated by HDG:
histogram of depth (hod),
histogram of depth gradients (hodg),
joint position differences (jpd),
joint movement volume features (jmv).
We follow  and evaluate the performance of the 10 variants of HDG in our experiments.
HPM+TM. We followed  and set the number of Fourier Pyramid levels to 3 and the number of low frequency Fourier coefficients to 4 for all datasets. We used the human pose model trained by Rahmani and Mian  to extract view-invariant features from each depth sequence. We also compared the recognition accuracies of this algorithm given Average Pooling (AP) versus Temporal Modelling (TM) used for extraction of CNN features.
P-LSTM. We applied the same normalization preprocessing step as in 
for the skeletal representation. In our experiments, the number of video segments and the number of hidden neurons for 1-layer RNN, 2-layer RNN, 1-layer LSTM, 2-layer LSTM, 1-layer P-LSTM and 2-layer P-LSTM were all set to 8 and 50, respectively. We also evaluated the performance of different numbers of video segments and different numbers of hidden neurons in P-LSTM. The learning rate and the number of epochs in our experiments were set to 0.01 and 300, respectively.
Clips+CNN+MTLN. The learning rate was set to 0.001 and the batch size was set to 100 for MTLN. We selected four different experimental settings from  to compare the performance of recognition: Frames+CNN, Clips+CNN+Pooling, Clips+CNN+Concatenation, and Clips+CNN+MTLN.
IndRNN. We used the Adam optimizer with the initial learning rate and applied the decay by 10 once the evaluation accuracy did not increase. For cross-subject and cross-view experiments, the dropout rates were set to 0.25 and 0.1, respectively.
ST-GCN. For the convolution operations, we used the optimal partition strategy according to the ablation study in . As different datasets have different numbers of body joints (see Table II), we reconstructed the spatio-temporal skeleton graphs. For NTU RGB+D dataset, we used the same experimental settings as described in  (e.g., we work with up to two human subjects per sequence). For the remaining 5 datasets, we used a different setting as only one performing subject was present per video.
Moreover, we performed extra experiments for IndRNN and ST-GCN: instead of using 3D skeleton sequences as inputs, we used jpd features which redefine the 3D skeleton joint positions by translating them to be centred at the torso (or ‘spine’) joint.
Iv-C Evaluation Measure
The recognition accuracy of an algorithm for any given action class is defined as the proportion of correct class labels returned by the algorithm:
To show the recognition accuracies of an algorithm for all the action classes, a confusion matrix is often used. The overall performance of an algorithm on a given dataset is evaluated using the average recognition accuracydefined as: , where is the total number of action classes in a given dataset.
To show the overall performance of each algorithm on datasets, we first rank the performance of each algorithm from 1 to 5 (a lower rank value represents a better performance) based on the recognition accuracy so each algorithm has its own rank value given the th dataset. We then compute the Average Rank (AVRank) as follows:
Iv-D Optimisation of Hyperparameters for HDG
There are 3 hyperparameters in the HDG algorithm. The first hyperparameter is the number of subvolumes, which we set to the same value as in. The second and third hyperparameters, which are the number of trees used in training and the threshold used in feature pruning, were optimised during our experiments (Table III). As the length of the combined features in the HDG algorithm is large (e.g., the length of the HDG-all features for the MSRAction3D dataset is 13,250), we trained one RDF to select the feature components that have high importance values. This helped to increase the processing speed without compromising the recognition accuracy.
We evaluated the effect of hyperparameters and on the HDG algorithm for different combinations of individual HDG features using the MSRAction3D dataset (for single-view) and the UWA3D Multiview Activity II dataset (for cross-view). Table III shows the optimal values for and obtained from the grid search. The corresponding dimensions of different HDG combined features before and after pruning are also indicated. Compared to other individual features of HDG, jpd and jmv are small-sized features, thus when used alone in both datasets, their dimensions are not reduced by much during feature pruning. However, when either or both of them are combined with other individual features in cross-view action recognition, their importance values are significantly higher. This allows a large reduction in feature dimension after pruning (see the last 4 rows of the table). Our experiments in the next section also confirm that skeleton-based features deal better with the view-invariance than depth-based features.
The optimal values of and shown in Table III were used to prune the HDG features for all datasets. As different datasets have different numbers of body joints (see Table II), the dimensions of these HDG features after pruning across datasets are not the same.
|Combination||MSRAction3D||UWA3D Multiview Activity II|
|features||before pruning||after pruning||before pruning||after pruning|
V Experimental Results
V-a MSRAction3D, 3D Action Pairs, CAD-60, and UWA3D Activity Datasets
|Method||MSRAction3D||3D Action Pairs||CAD-60||UWA3D Activity||AVRank|
|Hand-crafted features||HON4D  (Depth)||82.15||96.00||72.70||48.89||3.25|
|HOPC  (Depth)||85.49||92.44||47.55||60.58||3.25|
|LARP-SO-logarithm map  (Skel.)||88.69||92.96||69.12||51.96||–|
|LARP-SO-unwrapping while rolling  (Skel.)||88.47||94.09||69.12||53.05||–|
|LARP-SO-FTP  (Skel.)||89.40||94.67||76.96||50.41||2.50|
|HDG-hod  (Depth)||66.22||81.20||26.47||44.35||–|
|HDG-hodg  (Depth)||70.34||90.98||50.98||54.23||–|
|HDG-jpd  (Skel.)||55.54||53.78||46.08||40.88||–|
|HDG-jmv  (Skel.)||62.40||84.87||41.18||51.02||–|
|HDG-hod+hodg  (Depth)||71.81||90.96||51.96||55.17||–|
|HDG-jpd+jmv  (Skel.)||65.57||84.93||49.02||55.57||–|
|HDG-hod+hodg+jpd  (Depth + Skel.)||72.06||90.72||51.47||56.41||–|
|HDG-hod+hodg+jmv  (Depth + Skel.)||75.41||92.27||49.51||58.80||–|
|HDG-hodg+jpd+jmv  (Depth + Skel.)||75.00||92.28||52.94||59.82||–|
|HDG-all features  (Depth + Skel.)||75.45||92.13||51.96||60.33||4.00|
|SCK+DCK  (Skel.)||89.47||96.00||89.22||61.52||1.00|
|Deep learning features||Frames+CNN  (Skel.)||60.73||73.71||58.82||46.47||–|
|Clips+CNN+Pooling  (Skel.)||67.64||74.86||58.82||46.47||–|
|Clips+CNN+Concatenation  (Skel.)||71.27||78.29||61.76||53.85||–|
|Clips+CNN+MTLN  (Skel.)||73.82||79.43||67.65||54.81||2.25|
|HPM+AP  (Depth)||56.73||56.11||44.12||42.32||–|
|HPM+TM  (Depth)||72.00||98.33||44.12||54.78||2.50|
|1-layer RNN  (Skel.)||18.02||32.76||54.90||14.27||–|
|2-layer RNN  (Skel.)||27.80||56.13||54.91||35.36||–|
|1-layer LSTM  (Skel.)||62.26||67.14||61.77||50.81||–|
|2-layer LSTM  (Skel.)||65.33||73.72||63.24||46.78||–|
|1-layer P-LSTM  (Skel.)||70.50||70.86||61.76||55.16||–|
|2-layer P-LSTM  (Skel.)||69.35||72.00||67.65||50.81||2.75|
|Our implementation with modified hyperparam. values:|
|1-layer LSTM (8 segments, 100 hidden neurons) (Skel.)||64.75||73.14||58.82||52.58||–|
|2-layer P-LSTM (10 segments, 50 hidden neurons) (Skel.)||66.09||75.43||67.65||50.00||–|
|2-layer P-LSTM (10 segments, 100 hidden neurons) (Skel.)||67.43||71.43||54.41||50.32||–|
|2-layer P-LSTM (20 segments, 50 hidden neurons) (Skel.)||73.18||71.43||52.94||49.68||–|
|2-layer P-LSTM (20 segments, 100 hidden neurons) (Skel.)||70.50||71.43||58.82||53.55||–|
|IndRNN (4 layers)  (Skel.)||71.50||90.05||51.72||44.63||–|
|IndRNN (6 layers)  (Skel.)||72.91||89.53||57.03||42.66||–|
|Our improved results:|
|IndRNN (4 layers, with jpd) (Skel.)||76.34||82.66||84.69||52.09||–|
|IndRNN (6 layers, withjpd) (Skel.)||77.47||86.88||80.16||51.34||2.00|
|ST-GCN  (Skel.)||27.64 (69.09)||20.00 (77.14)||23.53 (70.59)||22.12 (45.83)||–|
|Our improved results:|
|ST-GCN (with jpd) (Skel.)||18.18 (64.00)||54.16 (96.57)||26.47 (67.65)||36.54 (70.51)||4.75|
*For ST-GCN, the numbers inside the parentheses denote the top-5 accuracy.
Table IV summarizes the results for the single-view action recognition on these datasets. The algorithms with the highest recognition accuracy for the handcrafted feature and deep learning feature categories are highlighted in bold.
Among the methods that use handcrafted features, SCK+DCK outperformed all other methods on all the datasets. The use of RBF kernels to capture higher-order statistics of the data and complexity of action dynamics demonstrates its effectiveness for action recognition. For the 3D Action Pairs and UWA3D Activity datasets, HON4D and HOPC are, respectively, the second top performers. The poorer performance of HDG was due to noise in the depth sequences which affected the HDG-hod and HDG-hodg features, even though the human subjects were successfully segmented. In general, HDG-hodg outperformed HDG-hod, and HDG-jmv outperformed HDG-jpd. We also found that concatenating more individual features in HDG (see Table IV, row HDG-hod+hodg+jpd to row HDG-all features) helped improve action recognition. We note that our results for HDG are different from those reported in  because we used cells  instead of cells to store the joint motion features.
For deep learning methods, the 1-layer P-LSTM method (8 video segments, 50 hidden neurons) outperformed others on the UWA3D Activity dataset. In general, a 1-layer LSTM with more hidden neurons performed better than a 1-layer LSTM with fewer neurons, and P-LSTM performed better than the traditional LSTM and RNN (see the last column in Table IV). The 2-layer P-LSTM (20 video segments, 50 hidden neurons) achieved the best recognition accuracy on the MSRAction3D dataset and HPM+TM outperformed on the 3D Action Pairs dataset. Comparing the last 5 rows of Table IV for different variants of 2-layer P-LSTM shows that having more video segments and/or hidden neurons does not guarantee better performance. The reasons are: (i) more video segments have less averaging effect and so it is likely that noisy video frames with unreliable skeletal information would be used for feature representation; (ii) having too many hidden neurons would cause overfitting in the training process.
Using the jpd features instead of the raw 3D joint coordinates boosts the recognition accuracies for both IndRNN and ST-GCN on almost all the datasets. For the CAD-60 and MSRAction3D datasets, the 4-layer IndRNN and 6-layer IndRNN using jpd features are the top two performers. Compared to using the raw 3D joint coordinates, the improvement due to the use of jpd is 4.56% for IndRNN (6 layers) on the MSRAction3D dataset and 32.97% for IndRNN (4 layers) on the CAD-60 dataset.
The last column of Table IV computed using Eq. (2) shows that SCK+DCK obtains average rank 1 score, followed closely by the 6-layer IndRNN with jpd with average rank 2 score. The ST-GCN method uses a more complex architecture having 9 layers of spatio-temporal graph convolutional operators, which require a large dataset for training. As all the datasets in Table IV are quite small, its poor performance (average rank 4.75 score) is not unexpected.
V-B NTU RGB+D Dataset
Table V summarizes the evaluation results for cross-subject and cross-view action recognition on the NTU RGB+D dataset, where methods are grouped into the handcrafted feature and deep learning feature categories. Top performing methods are highlighted in bold.
Among the methods using handcrafted features, SCK+DCK performed the best for both cross-subject action recognition and cross-view action recognition. Similarly to results on the four datasets shown in Table IV, combining more individual features in HDG resulted in higher recognition accuracy for both the cross-subject and cross-view action recognition.
Among deep learning methods, ST-GCN and the 6-layer IndRNN combined with the jpd features outperformed other deep learning methods in both cross-subject and cross-view action recognition. In particular, with jpd, ST-GCN became the top performer (achieving 83.36%) for cross-subject action recognition and IndRNN (6 layers) achieved the highest accuracy (89.0%) for cross-view action recognition. Compared to using the raw 3D skeleton joint coordinates, using the jpd features helps improve the recognition accuracies of both IndRNN and ST-GCN. For example, with jpd features, both the top-1 and top-5 accuracies of ST-GCN increased by 1.79% and 0.61% in the cross-subject experiment.
For the other deep learning methods, the recognition accuracies of 2-layer RNN, 2-layer LSTM and 2-layer P-LSTM are higher than those of 1-layer RNN, 1-layer LSTM and 1-layer P-LSTM. Similar to the results in Table IV, P-LSTM performed better than the traditional LSTM and RNN. HPM+AP and HPM+TM, on the other hand, did not perform so well. The reason is that both of these methods were trained given only 339 representative human poses from a human pose dictionary whereas the 60 action classes of NTU RGB+D dataset include many more human poses of higher complexity.
|Hand-crafted features||HON4D  (Depth)||30.6||7.3|
|HOPC  (Depth)||40.3||30.6|
|LARP-SO-FTP  (Skel.)||52.1||53.4|
|HDG-hod  (Depth)||20.1||13.5|
|HDG-hodg  (Depth)||23.0||25.2|
|HDG-jpd  (Skel.)||27.8||35.9|
|HDG-jmv  (Skel.)||38.1||50.0|
|HDG-hod+hodg  (Depth)||24.6||26.5|
|HDG-jpd+jmv  (Skel.)||39.7||51.9|
|HDG-hod+hodg+jpd  (Depth + Skel.)||29.4||38.8|
|HDG-hod+hodg+jmv  (Depth + Skel.)||39.0||57.0|
|HDG-hodg+jpd+jmv  (Depth + Skel.)||41.2||57.2|
|HDG-all features  (Depth + Skel.)||43.3||58.2|
|SCK+DCK  (Skel.)||72.8||74.1|
|Deep learning features||Frames+CNN  (Skel.)||75.7||79.6|
|Clips+CNN+Concatenation  (Skel.)||77.1||81.1|
|Clips+CNN+Pooling  (Skel.)||76.4||80.5|
|Clips+CNN+MTLN  (Skel.)||79.6||84.8|
|HPM+AP  (Depth)||40.2||42.2|
|HPM+TM  (Depth)||50.1||53.4|
|1-layer RNN  (Skel.)||56.0||60.2|
|2-layer RNN  (Skel.)||56.3||64.1|
|1-layer LSTM  (Skel.)||59.1||66.8|
|2-layer LSTM  (Skel.)||60.7||67.3|
|1-layer P-LSTM  (Skel.)||62.1||69.4|
|2-layer P-LSTM  (Skel.)||62.9||70.3|
|IndRNN (4 layers)  (Skel.)||78.6||83.8|
|IndRNN (6 layers)  (Skel.)||81.8||88.0|
|Our improved results:|
|IndRNN (4 layers, with jpd)(Skel.)||79.5||84.5|
|IndRNN (6 layers, with jpd)(Skel.)||83.0||89.0|
|ST-GCN  (Skel.)||81.57 (96.85)||88.76 (98.83)|
|Our improved results:|
|ST-GCN (with jpd) (Skel.)||83.36 (97.46)||88.84 (98.87)|
Our implementations for reproducing original authors’ experiment results.
For ST-GCN, the numbers inside the parentheses denote the top-5 accuracy.
V-C UWA3D Multiview Activity II Dataset
The ten algorithms were compared on the UWA3D Multiview Activity II dataset using cross-view action recognition. Table VI summarizes the results. The top 2 action recognition algorithms are highlighted in bold in each column for the handcrafted feature and deep learning feature categories.
For the methods using handcrafted features, the HDG-all features performed the best for cross-view action recognition (the last column in Table VI) followed by other HDG variants that use skeletons and depth. Among skeleton-only methods, HDG-jpd+jmv was the second best performer followed by SCK+DCK, HDG-jmv, and LARP-SO-FTP, which all performed better than depth-based features such as HON4D, HDG-hod, HDG-hodg and HDG-hod+hodg. According to the table, using one or both skeleton-based features (HDG-jpd and/or HDG-jmv) in HDG improved its results.
For deep learning methods, HPM+TM and HPM+AP achieved the highest results. Although Clips+CNN+MTLN performed better than other ‘Clips’ variants, it did not perform as good as HPM+TM and HPM+AP, due to the limited number of video samples in the dataset. P-LSTM performed better than the traditional LSTM and RNN. We noticed that stacking more layers for IndRNN or using jpd features for both IndRNN and ST-GCN did not help increase the recognition accuracy. This is due to the lack of representative training videos (compared to the results on the NTU RGB+D dataset in cross-view action recognition).
|Hand-crafted features||HON4D  (Depth)||31.1||23.0||21.9||10.0||36.6||32.6||47.0||22.7||36.6||16.5||41.4||26.8||28.9|
|HOPC  (Depth)||25.7||20.6||16.2||12.0||21.1||29.5||38.3||13.9||29.7||7.8||41.3||18.4||22.9|
|LARP-SO-logarithm map  (Skel.)||48.2||47.4||45.5||44.9||46.3||52.7||62.2||46.3||57.7||45.8||61.3||40.3||49.9|
|LARP-SO-unwrapping while rolling  (Skel.)||50.4||45.7||44.0||44.5||40.8||49.6||57.4||44.4||57.6||47.4||59.2||40.8||48.5|
|LARP-SO-FTP  (Skel.)||54.9||55.9||50.0||54.9||48.1||56.0||66.5||57.2||62.5||54.0||68.9||43.6||56.0|
|HDG-hod  (Depth)||22.5||17.4||12.5||10.0||19.6||20.4||26.7||13.0||18.7||10.0||27.9||17.2||18.0|
|HDG-hodg  (Depth)||26.9||34.2||20.3||18.6||34.7||26.7||41.0||29.2||29.4||11.8||40.7||28.8||28.5|
|HDG-jpd  (Skel.)||36.3||32.4||31.8||35.5||34.4||38.4||44.2||30.0||44.5||33.7||44.4||34.0||36.6|
|HDG-jmv  (Skel.)||57.2||59.3||59.3||54.3||56.8||50.6||63.4||52.4||65.7||53.7||67.7||56.9||58.1|
|HDG-hod+hodg  (Depth)||26.6||33.6||17.9||19.3||34.4||26.2||40.5||27.6||28.6||11.6||38.4||29.0||27.8|
|HDG-jpd+jmv  (Skel.)||61.0||61.8||59.3||56.0||60.0||57.4||68.8||54.2||71.1||57.2||69.7||59.0||61.3|
|HDG-hod+hodg+jpd  (Depth + Skel.)||31.0||43.5||25.7||21.4||45.9||31.1||53.2||35.7||38.0||11.6||49.7||38.3||35.4|
|HDG-hod+hodg+jmv  (Depth + Skel.)||59.0||62.2||58.1||52.0||62.5||57.1||66.0||54.2||67.7||52.7||70.3||61.1||60.2|
|HDG-hodg+jpd+jmv  (Depth + Skel.)||58.2||61.8||54.8||47.6||63.5||58.7||69.0||52.3||64.9||47.1||67.2||59.4||58.7|
|HDG-all features  (Depth + Skel.)||60.9||64.3||57.9||54.6||62.6||59.2||68.9||55.8||69.8||55.2||71.8||62.6||61.9|
|SCK+DCK  (Skel.)||56.0||61.8||50.0||61.8||54.1||56.7||71.4||56.9||72.9||49.6||71.7||44.0||58.9|
|Deep learning features||Frames+CNN  (Skel.)||30.4||29.8||27.8||23.8||31.7||27.2||39.2||27.8||31.4||23.2||38.8||28.6||30.0|
|Clips+CNN+Pooling  (Skel.)||31.6||33.3||30.6||27.8||35.3||29.6||44.3||31.3||35.3||23.2||43.1||31.0||33.0|
|Clips+CNN+Concatenation  (Skel.)||33.2||36.9||32.5||29.8||36.9||31.2||46.3||31.0||39.2||23.6||45.5||31.7||34.8|
|Clips+CNN+MTLN  (Skel.)||36.4||38.9||34.1||30.6||37.7||33.2||46.7||31.3||38.8||25.6||49.8||33.7||36.4|
|HPM+AP  (Depth)||68.3||51.7||60.2||62.2||38.7||50.0||58.7||37.8||70.6||61.6||74.0||55.6||57.5|
|HPM+TM  (Depth)||81.7||76.4||74.1||78.7||57.9||69.4||75.8||62.9||81.4||79.9||83.3||73.7||74.6|
|1-layer RNN  (Skel.)||11.4||11.1||10.0||14.9||11.9||13.3||11.1||15.2||10.3||12.2||9.8||12.4||12.0|
|2-layer RNN  (Skel.)||23.4||21.6||27.3||22.7||26.3||20.3||21.7||19.5||20.7||24.8||27.0||21.5||23.1|
|1-layer LSTM  (Skel.)||28.9||12.8||19.2||15.5||20.0||33.1||48.2||16.5||43.3||15.5||38.7||16.4||25.7|
|2-layer LSTM  (Skel.)||29.7||16.3||20.4||12.8||26.8||30.5||53.0||11.0||37.0||17.1||47.4||20.0||26.8|
|1-layer P-LSTM  (Skel.)||26.8||23.5||22.4||22.7||24.4||31.3||49.6||20.3||38.3||19.5||45.9||17.6||28.5|
|2-layer P-LSTM  (Skel.)||30.1||24.7||25.2||21.5||24.4||37.4||51.0||22.7||43.1||21.1||47.8||17.2||30.5|
|Our implementation with modified hyperparam. values:|
|2-layer P-LSTM (10 segments, 50 hidden neurons) (Skel.)||23.5||24.3||22.4||25.5||25.6||30.9||48.6||21.1||41.5||19.5||43.9||15.6||28.5|
|2-layer P-LSTM (10 segments, 100 hidden neurons) (Skel.)||21.1||22.7||24.8||21.1||22.4||33.7||47.8||21.1||45.1||22.0||49.4||17.6||29.1|
|2-layer P-LSTM (20 segments, 50 hidden neurons) (Skel.)||25.2||18.3||24.8||21.5||26.4||30.5||49.0||19.1||41.5||24.4||43.5||14.8||28.3|
|2-layer P-LSTM (20 segments, 100 hidden neurons) (Skel.)||27.6||24.3||24.8||21.9||28.4||32.9||44.3||24.3||44.3||20.3||45.9||15.6||29.6|
|IndRNN (4 layers)  (Skel.)||34.3||53.8||35.2||42.5||39.1||38.9||49.2||42.5||46.0||27.1||48.6||30.9||40.7|
|IndRNN (6 layers)  (Skel.)||30.7||47.2||32.2||36.0||38.8||35.4||44.5||37.9||40.6||23.9||39.2||25.2||36.0|
|IndRNN (4 layers, with jpd) (Skel.)||33.5||40.2||26.9||40.9||30.4||41.1||50.7||36.7||46.1||24.6||49.0||23.0||36.9|
|IndRNN (6 layers, with jpd) (Skel.)||29.4||36.3||23.9||38.2||26.0||36.8||45.4||33.0||41.2||20.5||44.7||18.6||32.8|
|ST-GCN  (Skel.)||36.4||26.2||20.6||30.2||33.7||22.4||43.1||26.6||16.9||12.8||26.3||36.5||27.6|
|ST-GCN (with jpd) (Skel.)||29.6||21.8||15.5||19.1||17.1||18.8||35.3||14.3||31.0||14.8||13.3||15.5||20.5|
*This result is obtained from  for comparison.
Fig. 3 shows a confusion matrix for the HDG-all representation on the UWA3D Multiview Activity II dataset when and views were used for training and was used for testing. According the figure, the algorithm was confused by a few actions which have similar motion trajectories. For example, one hand waving is similar to two hand waving and two hand punching; walking is very similar to irregular walking; bending and putting down have very similar appearance; sneezing and coughing are very similar actions.
Vi-a Single-view versus cross-view
Table VII summarizes the performance of each algorithm for both single-view action recognition and cross-view action recognition. One can observe from the table that some algorithms such as HON4D, LARP-SO-FTP, Clips+CNN+MTLN, P-LSTM, and SCK+DCK performed better for single-view action recognition but performed slightly worse in cross-view action recognition. For cross-subject action recognition, SCK+DCK outperformed all other algorithms; for cross-view action recognition, HDG-all, SCK+DCK and our improved IndRNN with jpd features all performed well (having the same average rank values).
Fig. 4 summarizes the average accuracy of all the algorithms grouped under the handcrafted and deep learning feature categories. On average, we found that the recognition accuracy for cross-view action recognition is lower than cross-subject action recognition using handcrafted features, with the poorest performance from depth-based features. Combining both depth- and skeleton-based features together helps improve cross-view action recognition and gives a similar performance for cross-subject action recognition. Overall, the performance on cross-view recognition is lower than cross-subject recognition, except for the deep learning depth-based feature methods. It should be noted that there are only two methods (HPM+AP and HPM+TM) falling into this category and they performed well on the UWA3D Multiview Activity II dataset and reasonably well on the NTU RGB+D dataset.
|HON4D  (Depth)||52.8 / [3.6]||18.1 / [5.0]|
|HDG-all  (Depth + Skel.)||56.6 / [3.8]||60.1 / [1.5]|
|HOPC  (Depth)||55.9 / [3.4]||41.4 / [4.0]|
|LARP-SO-FTP  (Skel.)||65.0 / [2.4]||54.7 / [3.0]|
|SCK+DCK  (Skel.)||78.7 / [1.0]||66.5 / [1.5]|
|HPM+TM  (Depth)||58.7 / [3.0]||64.0 / [3.0]|
|Clips+CNN+MTLN  (Skel.)||74.3 / [2.4]||60.6 / [3.0]|
|P-LSTM  (Skel.)||64.0 / [3.0]||50.4 / [4.0]|
|IndRNN (Skel.)||72.6 / [2.4]||62.1 / [2.0]|
|IndRNN with jpd (Skel.)||77.6 / [2.0]||60.8 / [1.5]|
|ST-GCN (Skel.)||52.4 / [4.2]||58.2 / [3.5]|
|ST-GCN with jpd (Skel.)||57.7 / [4.0]||54.7 / [3.0]|
For cross-subject action recognition, the performance of each algorithm is computed by averaging all the recognition accuracies / rank values on the MSRAction3D, 3D Action Pairs, CAD-60, UWA3D Activity and NTU RGB+D datasets.
For cross-view action recognition, they are computed over the UWA3D Multiview Activity II and NTU RGB+D datasets.
Vi-B Influence of camera views in cross-view evaluation
To analyze the influence of using different combinations of camera views in training and testing, we classified all the algorithms in Table VI into 3 feature types: depth-based, skeleton-based, and depth+skeleton-based representations. Scatter plots showing the performance of all the algorithms on these three feature types are given in Fig. 5, where the blue dots and red crosses denote, respectively, algorithms using handcrafted features and deep learning features.
According to plots in Fig. 5 the recognition accuracy was high when the left view and right view were used for training and front view for testing (see axis tick mark -). This is obvious as the viewing angle of the front view is between and , as shown in Fig. 2. However, it is surprising that the recognition accuracy was even slightly higher for - and - (see Figs. 5(b) and 5(c)).
One can also notice that for depth-based methods which use handcrafted features, the recognition accuracy dropped the most when the left view and top view were used for training, and the right view was used for testing. The main reason for this behavior is that the visual appearances of an action in and are different, and it is very difficult to find the same view-invariant features from these views. Other combinations of views, such as -, -, and -, also led to a lower performance (see the distribution of the blue dots in Fig. 5(a)).
Vi-C Depth-based features versus skeleton-based features
Cross-subject action recognition. It seems that for the cross-subject action recognition, skeleton-based features outperformed depth-based features for both the handcrafted and deep learning feature categories (Fig. 4(a)). However, it is surprising that adding the depth-based features to skeleton-based features actually led to a slight decrease in the action recognition accuracy. The main reason is the background clutter and noise (for instance, see Fig. 1) in the depth sequences, making the depth-based features less representative for robust action recognition.
On the other hand, some skeleton-based algorithms such as LARP-SO-FTP and its variants performed well in human action recognition because they only extracted reliable features from those human body joints which have high confidence values. The only exception is the HOPC algorithm which uses depth-based features and which performed better than skeleton-based features such as HDG-jpd+jmv in the cross-subject action recognition (see the average performance in the last column of Table IV
). One must note that the HOPC algorithm is different from other depth-based algorithms as it treats the depth images as a 3D pointcloud. Such an approach allows the HOPC algorithm to estimate the orientations of the local surface patches on the body of the human subject to more robustly deal with viewpoint changes.
Cross-view action recognition. In Fig. 5(b), for the cross-view action recognition, methods using handcrafted features (blue dots) performed better than those using deep learning features (red crosses) for skeleton-based features. For depth-based features (Fig. 5(a)), no clear conclusion can be drawn for the handcrafted versus deep learning features.
Comparing the handcrafted features (blue dots) in both Figs. 5(a) and 5(b), we can see that algorithms using skeleton-based features produced better results than depth-based features. This is evident from Fig. 5(b) as the blue dots are well above the red crosses, whereas in Fig. 5(a) the blue dots are more spread out and they occupy mainly the lower region of the plot. Our pruning process of the HDG algorithm also proved that skeleton-based features are more robust than the depth-based algorithms in the cross-view action recognition.
For the plot in Fig. 5(c), the algorithms in our experiments did not use deep learning to compute depth+skeleton features, thus no comparison on the performance between using handcrafted and deep learning features is possible.
Vi-D Handcrafted features versus deep learning features
As expected, our experiments confirm that deep learning methods (e.g., Clips+CNN+MTLN, IndRNN, and ST-GCN) performed better on large datasets such as the NTU RGB+D dataset, which has more than 56,000 video sequences, but achieved a lower recognition accuracy on the other smaller datasets. The main reason for this behavior is the reliance of deep learning methods on large amount of data for training to avoid overfitting (in contrast to handcrafted methods). However, most existing action datasets have small numbers of performing subjects and action classes, and very few camera views due to high cost of collecting/annotating data.
We also found that the performance of most handcrafted methods is highly dependent on the datasets. For example, HON4D and HOPC performed better only on some specific datasets, such as MSRAction3D and 3D Action Pairs, but achieved lower performance on the NTU RGB+D dataset. This means that features that have been handcrafted to work on one dataset may not be transferable to another dataset. Moreover, as handcrafted methods prevent overfitting well, they may lack the capacity to learn from big data. With larger benchmark datasets becoming available to the research community, future research trend is more likely to shift to using deep learning features with efforts being put into tuning their hyperparameters.
Vi-E ‘Quo Vadis, action recognition?’
Based on the methods evaluated in this paper, we see the following trends in human action recognition. Handcrafted features progressed from global descriptors (e.g., HON4D in 2013) to local descriptors (e.g., HOPC in 2016) and combinations of global and local feature representation (e.g., HDG in 2014). More recent works focus on designing robust 3D human joint-based representations (e.g., LARP-SO-FTP and SCK+DCK in 2016). Moreover, researchers have adopted features extracted from skeleton sequences as they are easier to process and analyze than depth videos. We also note that deep learning representations are evolving from basic neural networks (e.g., traditional RNN and LSTM) to adapted and/or dedicated networks that rely on a pre-trained network (e.g., HPM+AP, HPM+TM, and Clips+CNN+MTLN). Researchers also modify basic neural network models to improve their learning capability for action recognition (e.g., P-LSTM and IndRNN). Recently, new models such as ST-GCN were devised to robustly model the human actions in large datasets. Given the surge of large datasets and powerful GPU machines, the newest methods learn representations in an end-to-end manner.
We have presented, analyzed and compared ten state-of-the-art algorithms in 3D action recognition on six benchmark datasets. These algorithms cover the use of handcrafted and deep learning features computed from depth and skeleton video sequences. We believe that our comparison results will be useful for future researchers interested in designing robust representations for recognizing human actions from videos captured by the Kinect camera and/or similar action sequence capturing sensors.
In our study, we found that skeleton-based features are more robust than depth-based features for both cross-subject action recognition and cross-view action recognition. Handcrafted features performed better than deep learning features given smaller datasets. However, deep learning methods achieved very good results if trained on large datasets. While accuracy as high as 90% has been achieved by some algorithms on cross-subject action recognition, the average accuracy on cross-view action recognition is much lower.
From our literature review and comprehensive experiments, we see that many research papers on human action recognition have already attempted to tackle the challenging issues mentioned in the introduction of the paper. Examples include using a skeleton model or treating depth maps as 3D pointcloud to overcome the changes of viewpoint; using Fourier temporal pyramid or dynamic time warping to deal with different speeds of execution of actions; using depth videos to overcome changes in lighting conditions and eliminate visual appearances that may be irrelevant to the actions performed. While these papers all report promising action recognition results, new and more robust action recognition algorithms are still required, given that there is a pressing demand for using these algorithms in real and new environments.
We would like to thank the authors of HON4D , HOPC , LARP-SO , HPM+TM , IndRNN  and ST-GCN  for making their codes publicly available. We thank the ROSE Lab of Nanyang Technological University (NTU), Singapore, for making the NTU RGB+D dataset freely accessible. We would like to thank Data61/CSIRO for providing the computing cluster for running our experiments.
- Bobick and Davis  A. F. Bobick and J. W. Davis, “The Recognition of Human Movement Using Temporal Templates,” TPAMI, vol. 23, no. 3, pp. 257–267, March 2001.
- Dollar et al.  P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie, “Behavior Recognition via Sparse Spatio-Temporal Features,” ICCV, pp. 1–8, 2005.
- Blank et al.  M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri, “Actions as Space-Time Shapes,” ICCV, pp. 1–8, 2005.
- Ke et al.  Y. Ke, R. Sukthankar, and M. Hebert, “Spatio-temporal Shape and Flow Correlation for Action Recognition,” Proc. 7th Int. Workshop on Visual Surveillance, 2007.
- Laptev et al.  I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, “Learning Realistic Human Actions from Movies,” CVPR, pp. 1–8, 2008.
- Liu and Shah  J. Liu and M. Shah, “Learning Human Actions via Information Maximization,” CVPR, pp. 1–9, 2008.
- Bregonzio et al.  M. Bregonzio, S. Gong, and T. Xiang, “Recognising Actions as Clouds of Space-Time Interest Points,” CVPR, pp. 1948–1955, 2009.
- Liu et al.  J. Liu, J. Luo, and M. Shah, “Recognizing Realistic Actions from Videos in the Wild’,” CVPR, pp. 1–8, 2009.
- Li et al.  W. Li, Z. Zhang, and Z. Liu, “Action Recognition Based on A Bag of 3D Points,” in CVPR, 2010, pp. 9–14.
Yang and Tian 
X. Yang and Y. Tian, “EigenJoints-based Action Recognition Using Naive-Bayes-Nearest-Neighbor,” inCVPR, 2012, pp. 14–19.
- Oreifej and Liu  O. Oreifej and Z. Liu, “HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences,” in CVPR, 2013, pp. 716–723.
- Rahmani et al. [2014a] H. Rahmani, A. Mahmood, D. Q. Huynh, and A. Mian, “Real Time Action Recognition Using Histograms of Depth Gradients and Random Decision Forests,” in WACV, 2014, pp. 626–633.
- Rahmani et al. [2016a] H. Rahmani, A. Mahmood, D. Q. Huynh, and A. Mian, “Histogram of Oriented Principal Components for Cross-View Action Recognition,” TPAMI, pp. 2430–2443, December 2016.
- Rahmani and Mian  H. Rahmani and A. Mian, “3D Action Recognition from Novel Viewpoints,” CVPR, pp. 1–12, 2016.
- Mallick et al.  T. Mallick, P. P. Das, and A. K. Majumdar, “Characterizations of Noise in Kinect Depth Images: A Review,” IEEE SEN, vol. 14, no. 6, pp. 1731–1740, June 2014.
- Shotton et al.  J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, “Real-Time Human Pose Recognition in Parts from Single Depth Images,” in CVPR, 2011, pp. 1297–1304.
- Vemulapalli et al.  R. Vemulapalli, F. Arrate, and R. Chellappa, “Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group,” CVPR, pp. 588–595, 2014.
- Shahroudy et al. [2016a] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang, “NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis,” CVPR, pp. 1010–1019, 2016.
- Vemulapalli and Chellappa  R. Vemulapalli and R. Chellappa, “Rolling Rotations for Recognizing Human Actions from 3D Skeletal Data,” CVPR, pp. 4471–4479, 2016.
- Ke et al. [2017a] Q. Ke, S. An, M. Bennamoun, F. Sohel, and F. Boussaid, “SkeletonNet: Mining Deep Part Features for 3D Action Recognition,” IEEE SPL, vol. 24, no. 6, pp. 731–735, 2017.
- Ke et al. [2017b] Q. Ke, M. Bennamoun, S. An, F. Sohel, and F. Boussaid, “A New Representation of Skeleton Sequences for 3D Action Recognition,” in CVPR, June 2017.
- Rahmani and Bennamoun  H. Rahmani and M. Bennamoun, “Learning Action Recognition Model From Depth and Skeleton Videos,” in ICCV, 2017, pp. 5832–5841.
- Koniusz et al.  P. Koniusz, A. Cherian, and F. Porikli, “Tensor Representations via Kernel Linearization for Action Recognition from 3D Skeletons,” ECCV, pp. 1–19, 2016.
- Zhang et al.  J. Zhang, H. P. H. Shum, J. Han, and L. Shao, “Action Recognition From Arbitrary Views Using Transferable Dictionary Learning,” in TIP, 2018, pp. 4709–4723.
- Rahmani et al.  H. Rahmani, A. Mian, and M. Shah, “Learning a Deep Model for Human Action Recognition from Novel Viewpoints,” TPAMI, pp. 1–14, 2015.
- Yan et al.  S. Yan, Y. Xiong, and D. Lin, “Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition,” in AAAI, 2018.
- Li et al. [2018a] S. Li, W. Li, C. Cook, C. Zhu, and Y. Gao, “Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN,” CVPR, pp. 5457–5466, 2018.
- Li et al. [2018b] C. Li, Z. Cui, W. Zheng, C. Xu, and J. Yang, “Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition,” in AAAI, 2018, pp. 3482–3489.
- Wang and Wang  H. Wang and L. Wang, “Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks,” in CVPR, 2017, pp. 1–10.
Tang et al. 
Y. Tang, Y. Tian, J. Lu, P. Li, and J. Zhou, “Deep Progressive Reinforcement Learning for Skeleton-based Action Recognition,” inCVPR, 2018, pp. 1–10.
- Liu et al.  J. Liu, G. Wang, P. Hu, L.-Y. Duan, and A. C. Kot, “Global Context-Aware Attention LSTM Networks for 3D Action Recognition,” in CVPR, 2017, pp. 1–10.
- Huang et al.  Z. Huang, C. Wan, T. Probst, and L. V. Gool, “Deep Learning on Lie Groups for Skeleton-based Action Recognition,” in CVPR, 2017, pp. 6099–6108.
- Lee et al.  I. Lee, D. Kim, S. Kang, and S. Lee, “Ensemble Deep Learning for Skeleton-based Action Recognition using Temporal Sliding LSTM networks,” in ICCV, 2017, pp. 1012–1020.
- Si et al.  C. Si, Y. Jing, W. Wang, L. Wang, and T. Tan, “Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning,” in ECCV, 2018, pp. 1–16.
- Wang and Wang  H. Wang and L. Wang, “Beyond Joints: Learning Representations From Primitive Geometries for Skeleton-Based Action Recognition and Detection,” in TIP, 2018, pp. 4382–4394.
- Liu et al.  J. Liu, A. Shahroudy, D. Xu, A. C. Kot, and G. Wang, “Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates,” in TPAMI, 2018, pp. 1–14.
- Tanfous et al.  A. B. Tanfous, H. Drira, and B. B. Amor, “Coding Kendall’s Shape Trajectories for 3D Action Recognition,” in CVPR, 2018, pp. 2840–2849.
- Zheng et al.  N. Zheng, J. Wen, R. Liu, L. Long, J. Dai, and Z. Gong, “Unsupervised Representation Learning with Long-Term Dynamics for Skeleton Based Action Recognition,” in AAAI, 2018, pp. 2644–2651.
- Fishkin et al.  K. P. Fishkin, M. Philipose, and A. Rea, “Hands-On RFID: Wireless Wearables for Detecting Use of Objects,” IEEE ISWC, 2005.
- Hodges and Pollack  M. R. Hodges and M. E. Pollack, “An ‘Object-Use Fingerprint’: The Use of Electronic Sensors for Human Identification,” UbiComp, 2007.
- Buettner et al.  M. Buettner, R. Prasad, M. Philipose, and D. Wetherall, “Recognizing Daily Activities with RFID-Based Sensors,” UbiComp, 2009.
- Sung et al.  J. Sung, C. Ponce, B. Selman, and A. Saxena, “Human Activity Detection from RGBD Images,” PAIR, 2011.
- Sung et al.  J. Sung, C. Ponce, B. Selman, and A. Saxena, “Unstructured Human Activity Detection from RGBD Images,” in ICRA, 2012.
- Koppula et al.  H. S. Koppula, R. Gupta, and A. Saxena, “Learning Human Activities and Object Affordances from RGB-D Videos,” IJRR, Jan 2013.
- Gupta et al.  S. Gupta, R. Girshick, P. Arbelaez, and J. Malik, “Learning Rich Features from RGB-D Images for Object Detection and Segmentation,” ECCV, pp. 1–16, 2014.
- Bilen et al.  H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, and S. Gould, “Dynamic Image Networks for Action Recognition,” CVPR, pp. 3034–3042, 2016.
- Shahroudy et al. [2016b] A. Shahroudy, T.-T. Ng, Q. Yang, and G. Wang, “Multimodal Multipart Learning for Action Recognition in Depth Videos,” TPAMI, vol. 38, no. 10, pp. 2123–2129, 2016.
- Feichtenhofer et al.  C. Feichtenhofer, A. Pinz, and R. P. Wildes, “Spatiotemporal Multiplier Networks for Video Action Recognition,” CVPR, pp. 4768–4777, 2017.
- Kar et al.  A. Kar, N. Rai, K. Sikka, and G. Sharma, “AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos,” CVPR, pp. 3376–3385, 2017.
- Cherian et al. [2017a] A. Cherian, P. Koniusz, and S. Gould, “Higher-order Pooling of CNN Features via Kernel Linearization for Action Recognition,” WACV, pp. 130–138, 2017.
- Cherian et al. [2017b] A. Cherian, B. Fernando, M. Harandi, and S. Gould, “Generalized Rank Pooling for Activity Recognition,” CVPR, pp. 3222–3231, 2017.
- Shahroudy et al.  A. Shahroudy, T.-T. Ng, Y. Gong, and G. Wang, “Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos,” TPAMI, pp. 1045–1058, 2018.
- Carreira and Zisserman  J. Carreira and A. Zisserman, “Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset ,” in CVPR, 2017, pp. 1–10.
- Feichtenhofer et al.  C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convolutional Two-Stream Network Fusion for Video Action Recognition,” in CVPR, 2016, pp. 1–9.
- Hu et al.  J.-F. Hu, W.-S. Zheng, J. Pan1, J. Lai, and J. Zhang, “Deep Bilinear Learning for RGB-D Action Recognition,” in ECCV, 2018, pp. 1–17.
- Wang et al.  J. Wang, Z. Liu, Y. Wu, and J. Yuan, “Mining Actionlet Ensemble for Action Recognition with Depth Cameras,” CVPR, pp. 1290–1297, 2012.
- Xia and Aggarwal  L. Xia and J. K. Aggarwal, “Spatio-Temporal Depth Cuboid Similarity Feature for Activity Recognition Using Depth Camera,” in CVPR, 2013, pp. 2834–2841.
- Rahmani et al. [2014b] H. Rahmani, A. Mahmood, D. Q. Huynh, and A. Mian, “HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognition,” in ECCV, 2014, pp. 742–757.
- Rahmani et al. [2014c] H. Rahmani, A. Mahmood, D. Q. Huynh, and A. Mian, “Action Classification with Locality-constrained Linear Coding,” in ICPR, 2014, pp. 3511–3516.
- Yang and Tian  X. Yang and Y. Tian, “Super Normal Vector for Activity Recognition using Depth Sequences,” in CVPR, 2014, pp. 804–811.
- Wang et al.  P. Wang, W. Li, Z. Gao, J. Zhang, C. Tang, and P. Ogunbona, “Deep Convolutional Neural Networks for Action Recognition Using Depth Map Sequences,” in CVPR, 2015, pp. 1–8.
- Wang et al.  P. Wang, W. Li, Z. Gao, J. Zhang, C. Tang, and P. O. Ogunbona, “Action Recognition From Depth Maps Using Deep Convolutional Neural Networks,” IEEE T HUM-MACH SYST, pp. 498–509, 2016.
- Rahmani et al. [2016b] H. Rahmani, D. Q. Huynh, A. Mahmood, and A. Mian, “Discriminative Human Action Classification using Locality-constrained Linear Coding,” Pattern Recognit. Lett., pp. 62–71, 2016.
- Zhang et al.  C. Zhang, Y. Tian, X. Guo, and J. Liu, “DAAL: Deep Activation-based Attribute Learning for Action Recognition in Depth Videos,” CVIU, pp. 37–49, 2017.
- Shi and Kim  Z. Shi and T.-K. Kim, “Learning and Refining of Privileged Information-based RNNs for Action Recognition from Depth Sequences,” in CVPR, 2017, pp. 3461–3470.
- Xia et al.  L. Xia, C.-C. Chen, and J. K. Aggarwal, “View Invariant Human Action Recognition Using Histograms of 3D Joints,” CVPRW, 2012.
- Du et al.  Y. Du, W. Wang, and L. Wang, “Hierarchical Recurrent Neural Network for Skeleton Based Action Recognition,” in CVPR, 2015, pp. 1110–1118.
- Veeriah et al.  V. Veeriah, N. Zhuang, and G.-J. Qi, “Differential Recurrent Neural Networks for Action Recognition,” CVPR, pp. 1–9, 2015.
- Zhu et al.  W. Zhu, C. Lan, J. Xing, W. Zeng, Y. Li, L. Shen, and X. Xie, “Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks,” AAAI, 2016.
- Liu et al.  J. Liu, A. Shahroudy, D. Xu, and G. Wang, “Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition,” ECCV, pp. 816–833, 2016.
- Amor et al.  B. B. Amor, J. Su, and A. Srivastava, “Action Recognition Using Rate-Invariant Analysis of Skeletal Shape Trajectories,” TPAMI, vol. 38, no. 1, pp. 1–13, 2016.
- Elmadany et al.  N. E. D. Elmadany, Y. He, and L. Guan, “Information Fusion for Human Action Recognition via Biset/Multiset Globality Locality Preserving Canonical Correlation Analysis,” in TIP, 2018, pp. 5275–5287.
- Si et al.  C. Si, W. Chen, W. Wang, L. Wang, and T. Tan, “An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition,” in CVPR, 2019, pp. 1227–1236.
- Li et al.  M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, and Q. Tian, “Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition,” in CVPR, 2019, pp. 3595–3603.
- Shi et al. [2019a] L. Shi, Y. Zhang, J. Cheng, and H. Lu, “Skeleton-Based Action Recognition with Directed Graph Neural Networks,” in CVPR, 2019, pp. 7912–7921.
- Shi et al. [2019b] L. Shi, Y. Zhang, J. Cheng, and H. Lu, “Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition,” in CVPR, 2019, pp. 12 027–12 035.
- Tas and Koniusz  Y. Tas and P. Koniusz, “CNN-based Action Recognition and Supervised Domain Adaptation on 3D Body Skeletons via Kernel Feature Maps,” in BMVC, 2018.
- Hochreiter and Schmidhuber  S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Comput., pp. 1735–1780, 1997.
- Gao et al.  Y. Gao, O. Beijbom, N. Zhang, and T. Darrell, “Compact Bilinear Pooling,” CVPR, pp. 1–10, 2016.
- Simonyan and Zisserman  K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-scale Image Recognition,” ICLR, pp. 1–14, 2015.
- Caruana  R. Caruana, Multitask Learning, ser. 1. Springer, 1997, vol. 28, no. 41-75.
- Bruna et al.  J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun, “Spectral Networks and Deep Locally Connected Networks on Graphs,” in ICLR, 2014, pp. 1–14.
- Defferrard et al.  M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering,” in NIPS, 2016, pp. 1–9.
- Kipf and Welling  T. N. Kipf and M. Welling, “Semi-Supervised Classification with Graph Convolutional Networks,” in ICLR, 2017, pp. 1–14.
- Koniusz et al.  P. Koniusz, L. Wang, and A. Cherian, “Tensor Representations for Action Recognition,” CoRR, 2019.
- Wang  L. Wang, “Analysis and Evaluation of Kinect-based Action Recognition Algorithms,” Master’s thesis, School of the Computer Science and Software Engineering, The University of Western Australia, Nov 2017.