Human action recognition has numerous applications in smart surveillance, human-computer interaction, sports and elderly care[1, 2]. Kinect like depth cameras have become popular for this task because depth sequences are somewhat immune to variations in illumination, clothing color and texture. However, the presence of occlusions, sensor noise, variations in action execution speed and most importantly sensor viewpoint still make action recognition challenging.
Designing an efficient representation for 3D video sequences is an important task for many computer vision problems. Most existing techniques (e.g.[3, 4, 5, 6, 7]) treat depth sequences similar to conventional videos and use color-based action recognition representations. However, simple extensions of color based action recognition techniques to depth sequences are not optimal[8, 9]. Instead of processing depth sequences, richer geometric features can be extracted from 3D pointcloud videos.
Action recognition research[10, 4, 11, 12, 8, 9, 5, 6, 7, 13, 14] has mainly focused on actions captured from a fixed viewpoint. However, a practical human action recognition system should be able to recognize actions from different views. Some view-invariant approaches[15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28] have also been proposed for cross-view action recognition where recognition is performed from an unknown and/or unseen view. These approaches generally rely on geometric constraints[15, 16, 17, 18, 19], view-invariant features[20, 21, 22, 23, 24, 25, 26], and human body joint tracking[27, 28]. More recent approaches transfer features across views[29, 30, 31, 32, 33, 34, 35, 36]. However, these methods do not perform as good as fixed view action recognition. The majority of cross-view action recognition research has focused on color videos or skeleton data. Cross-view action recognition from 3D videos remains an under explored area. We believe that cross-view action recognition from 3D pointcloud videos holds more promise because view-invariant features can be extracted from such videos.
We approach the cross-view action recognition problem from a novel perspective by directly processing the 3D pointcloud sequences (Fig. 1). We extend our previous research where we proposed a new descriptor, the Histogram of Oriented Principal Components (HOPC), to capture the local geometric characteristics around each point in a 3D pointcloud sequence. Based on HOPC, we propose a Spatio-Temporal Keypoint (STK) detection method so that view-invariant Local HOPC descriptors are extracted from the most discriminative points within a sequence of 3D pointclouds. We also propose another descriptor, STK-D, which is computed from the spatio-temporal distribution of the STKs. Since Local HOPC and STK-D capture complementary information, their combination significantly improves the cross-view action recognition accuracy over existing state-of-the-art.
To achieve view invariance for HOPC, all points within an adaptable spatio-temporal support volume of each STK are aligned along the eigenvectors of its spatial support volume. In other words, the spatio-temporal support volume is aligned in a local object centered coordinate basis. Thus, HOPC descriptor extracted from this aligned support volume is view-invariant (Fig. 2). Note that this strategy does not necessarity work for other descriptors as shown in Fig. 2. As humans often perform the same action at different speeds, for speed invariance, we propose automatic temporal scale selection that minimizes the eigenratios over a varying temporal window size independently at each STK.
Our four main contributions are summarized as follows: Firstly, we propose the HOPC descriptor which encodes shape and motion in a robust way. Secondly, we propose a view-invariant Spatio-Temporal Keypoint (STK) detector that is integrated with HOPC in the sense that it detects points that are suitable for HOPC. Thirdly, we propose a global action descriptor based on the spatio-temporal distribution of STKs. Finally, we propose a method for viewpoint and speed invariant action recognition. Moreover, we introduce a new UWA3D Multiview Activity II dataset in addition to  which contains actions performed by subjects from four different views. This dataset is larger in number of action classes than existing 3D action datasets.
The proposed descriptors have been evaluated on two multi-view and three single-view human action recognition datasets. The former includes the Northwestern-UCLA Multiview Action3D and the UWA3D Multiview Activity II datasets whereas the latter includes MSR Action3D, MSR Daily Activity3D, and MSR Gesture3D datasets. Our extensive experimental results show that the proposed descriptors achieved significantly better accuracy compared to the nine existing state-of-the-art techniques[9, 10, 59, 3, 60, 27, 36, 28, 29].
2 Related Work
Based on the data type, action recognition methods can be divided into three categories including color-based, skeleton-based and depth-based methods. In color videos, a significant portion of the existing work has been proposed for single-view action recognition, where the training and test videos are captured from the same view. In order to recognize actions across different views, one approach is to collect data from all possible views and train a separate classifier for each view. However, this approach does not scale well due to the requirement of a large number of labeled samples for each view and it becomes infeasible as the number of action categories increases. To overcome this problem, some techniques infer 3D scene structure and use geometric transformations to achieve view invariance[15, 16, 17, 18, 19]. These methods critically rely on accurate detection of the body joints and contours, which are still open problems in real-world settings. Other methods focus on spatio-temporal features which are inherently view-invariant[20, 21, 22, 23, 24, 25, 26]. However, these methods have limitations as some of them require access to mocap data while others compromise discriminative power to achieve view invariance.
have become popular. These methods find a view independent latent space in which features extracted from different views are directly comparable. Such methods are either not applicable or perform poorly when the recognition is performed on videos from unknown and, more importantly, from unseen views. To overcome this problem, Wang et al. proposed cross-view action representation by exploiting the compositional structure in spatio-temporal patterns and geometrical relations among views. Although their method can be applied to action recognition from unknown and unseen views, it requires 3D skeleton data for training which is not always available. Our proposed approach also falls in this category except that it uses 3D pointcloud sequences and does not require skeleton data. To the best of our knowledge, we are the first to propose cross-view action recognition using 3D pointcloud videos.
In skeleton-based action recognition methods, multi-camera motion capture (MoCap) systems  have been used for human action recognition. However, such specialized equipment is marker-based and expensive. On the other hand, some other methods[59, 39, 28, 13, 27, 14] use the human joint positions extracted by the OpenNI tracking framework. For example, Yang and Tian used pairwise 3D joint position differences in each frame and temporal differences across frames to represent an action. Since 3D joints cannot capture all the discriminative information, the action recognition accuracy is compromised. Wang et al.  extended this approach by computing the histogram of occupancy patterns of a fixed region around each joint in each frame. In order to make this method more robust to viewpoint variations, they proposed a global orientation normalization using the skeleton data. In this method, a plane is fitted to the joints and a rotation matrix is computed to rotate this plane to the -plane. However, this method is only applicable if the subject is in an upright pose. Moreover, when the subject is in a non-frontal view, the joint positions may have large errors, making the normalization process unreliable. In contrast, our proposed orientation normalization method does not need the joint positions and can efficiently work in non-frontal as well as non upright positions. In addition to that, as our method performs local orientation normalization at each STK, it is more robust than the single global normalization proposed by.
Many of the existing depth-based action recognition methods use global features such as silhouettes and space-time volume information. For example, Li et al.  sampled boundary pixels from 2D silhouettes as a bag of features. Yang et al.  added temporal derivatives of 2D projections to get Depth Motion Maps (DMM). Vieira et al.  computed silhouettes in 3D by using the space-time occupancy patterns. Oreifej and Liu extended histogram of oriented 3D normals  to 4D by adding the time derivative. Recently, Yang and Tian extended HON4D by concatenating the 4D normals in the local neighbourhood of each pixel as its descriptor. Our proposed HOPC descriptor is more informative than HON4D because it captures the spread of data in three principal directions. Holistic methods may fail in scenarios where the subject significantly changes her/his spatial position[10, 9]. Some other methods use local features where a set of interest points are extracted from the depth sequence and a local feature descriptor is computed for each interest point. For example, Cheng et al. used the Cuboid interest point detector and proposed a Comparative Coding Descriptor (CCD). Due to the presence of noise in depth sequences, simply extending color-based interest point detectors, such as Cuboid, 3D Hessian and 3D Harris, degrades the efficiency and effectiveness of these detectors as most interest points are detected at irrelevant locations[8, 9].
Motion trajectory based action recognition methods[49, 50, 51, 20] are also not reliable in depth sequences. Therefore, recent depth based action recognition methods resorted to alternative ways to extract more reliable interest points. Wang et al. proposed Haar features to be extracted from each random subvolume. Xia and Aggarwal proposed a filtering method to extract spatio-temporal interest points. Their approach fails when the action execution speed is faster than the flip of the signal caused by sensor noise. Moreover, both techniques are not robust to viewpoint variations.
3 HOPC: Histogram of Oriented Principal Components
HOPC is extracted at each point within a sequence of 3D pointclouds , where denotes the number of 3D pointclouds in the sequence and is the 3D pointcloud at time . Consider a point in . We define two different support volumes for : a spatial support volume and a spatio-temporal support volume. The spatial support volume of , denoted by , contains the 3D points in that are in a sphere of radius centered at (Fig. 3(b)). To define the spatio-temporal support volume of , denoted by , we merge the sequence of pointclouds in the small time interval . The 3D points which are in a sphere of radius centered at are considered as (Fig. 3(c)).
The covariance matrix of the points is given by:
and denotes the number of points in the support volume of . Performing eigen decomposition on the covariance matrix gives us:
is a diagonal matrix containing the eigenvaluesof and contains the three corresponding orthonormal eigenvectors.
The HOPC descriptor is built by projecting each eigenvector onto directions obtained from the vertices of a regular polyhedron. In particular, we consider a regular dodecahedron which is composed of vertices, each of which corresponds to a histogram bin. Let be the vertices of a regular dodecahedron and let . For a regular dodecahedron centered at the origin, these vertices are given as:
8 vertices from
4 vertices from
4 vertices from
4 vertices from
where is the golden ratio.
Each eigenvector is a direction in the 3D space representing the distribution of point positions in the support volume. Therefore, its orientation has a
ambiguity. To resolve this orientation ambiguity, we consider the distribution of point vector directions and their magnitudes within the support volume of. That is, for each point , we compute and we determine the sign of each eigenvector as follows:
where the ‘sign’ function returns the sign of an input number. Note that the squared projection operation ensures that small projected values, which are often due to noise, are suppressed. If the signs of eigenvectors and disagree, i.e. , we switch the sign of the eigenvector whose value is the smallest. We then project each eigenvector onto to give us:
If perfectly aligns with , it should vote into only the bin. However, as the ’s are not orthogonal to each other, will have non-zero projection values in other bins as well. To overcome this effect, we quantize the projection values of by imposing a threshold value computed as follows:
where and are any two neighbouring vectors in . The quantized vector is then given by
where denotes a bin number. For the eigenvector, we define to be scaled by their corresponding eigenvalue :
We concatenate the histograms of oriented principal components of the three eigenvectors in decreasing order of magnitudes of their associated eigenvalues to form a descriptor for point :
The spatial HOPC descriptor encodes the shape of the support volume around . On the other hand, the spatio-temporal HOPC descriptor
encodes information from both shape and motion. Since the smallest principal component of the local surface is the total least squares estimate of the surface normal, the surface normals encoded in our descriptor are more robust to noise than gradient-based surface normals used in[45, 9]. Moreover, HOPC additionally encodes the first two eigenvectors which are more dominant compared to the third one. The computation of the spatial and spatio-temporal HOPC descriptors is shown in Fig. 3(d).
4 Spatio-Temporal Keypoint (STK) Detection
The aim of the STK detection is to find points in 3D pointcloud action sequences that satisfy three constraints:
Repeatability: STKs should be identified with high repeatability in different samples of the same action in the presence of noise and viewpoint changes.
Uniqueness: A unique coordinate basis should be obtained from the neighbourhood of the STKs for the purpose of view-invariant description.
Significant spatio-temporal variation: STKs should be detected where the neighbourhood has significant space-time variations.
To achieve these aims, we propose an STK detection technique which has high repeatability, uniqueness and detects points where space-time variation is significant. Consider a point within a sequence of 3D pointclouds. We perform eigen decomposition on the spatial and the spatio-temporal covariance matrices and as described in Section 3. For the first two constraints, we define the following ratios:
For 3D symmetrical surfaces, the ratio between the first two eigenvalues or last two eigenvalues are very close to . The principal components at such locations are, therefore, ambiguous. Thus, for a point to be qualified as a potential keypoint, the condition
must be satisfied, where is a small margin to cater for noise. This process eliminates ambiguous points and produces a subset of candidate keypoints which can be described uniquely in a local coordinate basis.
Recall that in (7) represents the spatial HOPC and the spatio-temporal HOPC at point . For the third constraint, a quality factor is computed for all candidate keypoints:
When , the quality factor is at the minimum value of which basically means that the candidate point has a stationary spatio-temporal support volume. On the other hand, significant variations in space-time change the direction and magnitude of spatio-temporal eigenvectors with respect to the spatial eigenvectors. Thus, is large when a significant motion occurs in the spatio-temporal support volume.
STKs that are in the vicinity of each other are similar as they describe more or less the same local support volume. We perform a non-maximum suppression to keep a minimum distance between STKs. We define radius (with ) and time interval (with ) where is the frame number being considered. The candidate STKs are firstly sorted according to their quality values. Starting from the highest quality STK, all candidate STKs falling within and from it are discarded. The same process is repeated on the remaining candidate STKs until only a desired number, , of STKs are left. Figure. 3 shows the steps of our STK detection algorithm. Figure. 4 shows the extracted STKs from four different views for a 3D pointcloud sequence corresponding to the two hand waving action.
5 View-Invariant STK Description (Local HOPC)
The HOPC descriptor discussed in Section 3 is not view-invariant yet. We compute Local HOPC only at the STKs since it is possible to normalize the orientation of the local region only at these points i.e. a unique local coordinate basis can only be defined at these points. We perform orientation normalization at each STK using the eigenvectors of its spatial covariance matrix (see Section 3). We consider the eigenvectors of as a local object centered coordinate basis. Note that the matrix is orthonormal and can be used as a valid 3D rotation matrix, since:
We apply the 3D rotation to all the mean-centered points within the spatio-temporal support volume of , , and bring them to a canonical coordinate system:
where denotes the rotated point in the local object centered coordinate basis. Note that the first, second, and third principal components are now aligned with the axes of the Cartesian coordinates. Since the same STKs in two different views have the same canonical representation, we can do cross-view keypoint matching (Fig. 2). It is important to note that our STK detection algorithm has already pruned ambiguous points to make the local object centered coordinate basis unique, i.e. no two eigenvectors have the same eigenvalues. Therefore, the eigenvector with the maximum eigenvalue will always map to the axis, the second largest to the axis and the smallest to the axis.
After the orientation normalization given in (12), for each point , we inspect the eigenratios and (Eq. (9)) computed using neighbouring points of to determine how the HOPC descriptor at should contribute to the STK descriptor computation of . When and are both larger than , where is a small margin, all the eigenvectors are uniquely defined and, therefore, can all contribute to the STK descriptor. When and , the first two eigenvectors are ambiguous and so only the third eigenvector should contribute to the STK descriptor. A similar argument applies to the case where and . When both and , then has no contribution to the descriptor computation. In summary, the following three criteria need to be considered for the construction of :
If & , ;
If & , ;
If & , .
Next, the orientation normalized spatio-temporal support volume around the STK is partitioned into spatio-temporal cells along the , , and dimensions. We use , to denote the cell. The cell descriptor is computed by accumulating the ’s
and then normalizing
We define the final view-invariant descriptor, , of STK to be the concatenation of obtained from all the cells:
The above steps are repeated for all the STKs. Thus, the STK descriptors encode view-invariant spatio-temporal patterns that will be used for action description.
6 Action Description
6.1 Bag of STK Descriptors
We represent each sequence of 3D pointclouds by a set of STK descriptors. Inspired by the successful bag-of-words approach for object recognition, we build a codebook by clustering the STK descriptors (
) with the K-means algorithm. Clustering is performed over all action descriptors extracted from all training view samples. Thus, the codebook that we learn is not single action or single view specific. For a fair evaluation, we do not use the target test views in codebook learning or any other training task. We consider each cluster as a codeword that represents a specific spatio-temporal pattern shared by the STKs in that cluster. One codeword is assigned to each STK descriptor based on the minimum Euclidean distance. The histogram of codewords is used as an action descriptor. For classification, we use an SVM classifier with the histogram intersection kernel.
6.2 Mining Discriminative Codebooks
Not all codewords have the same level of discrimination. Some codewords may encode movements that do not offer good discrimination among different actions, e.g. the sway of the human body. We use the F-score to find the most discriminative features in the codebook and discard non-discriminative features. The F-score
measures the discrimination of two sets of real numbers. For more than two sets of real numbers, we use the multiset F-score to measure their discrimination. Given the training histogram of codewords , and action classes, if the number of the samples in the th () class is , then the F-score of the th histogram bin is defined as:
where and are the average of the th histogram bin of all samples and the th class samples, respectively, and is the th histogram bin of the th sample in the th class. The larger is the F-score, the more discriminative is the corresponding histogram bin. Therefore, we rank the codewords by their F-scores and select the codewords whose F-scores are higher than a threshold. In our experiments, up to improved accuracy was observed by selecting the top discriminative features out of the total .
6.3 Encoding Spatio-Temporal STK Distribution
The bag-of-words approach efficiently encodes the local spatio-temporal information in a 3D pointcloud sequence. However, it ignores the spatio-temporal relationship among the STKs. We observed that encoding the distribution of STKs in space-time (Fig. 4) can further improve the discrimination between different actions in addition to the bag-of-words based descriptors. To incorporate the space-time positional information of STKs, we propose a method that encodes this information.
Let represent the set of all selected STKs within a sequence of 3D pointclouds , where is the number of STKs and are the coordinates of an STK in the 4D space with and being the spatial coordinates, being depth and
being time. To cope with the heterogeneity in the vectors, we normalize the vectors so that all their components have zero-mean and unit variance.
To simplify the description, let us assume that the set now have all the normalized vectors as described above. By dropping the time axis, we have a set of normalized 3D STKs: . Eigen decomposition is then applied to the covariance matrix of points in to yield two eigenratios and . To get a unique coordinate basis, we require that , where is a small constant. If these constraints are not satisfied, we perform an iterative refinement of STKs as follows. Given initial STKs, in each iteration, (where ) STKs with the lowest quality factor (Eq. (10)) are removed. Eigen decomposition is applied to the remaining points to yield two new eigenratios. This process is iterated until the eigenratio constraints are satisfied. Generally, three iterations are sufficient.
To achieve a view-invariant representation, all points in are aligned along , i.e., for all , we set where is the eigenvector matrix obtained from the eigen decomposition in the last iteration. The normalized temporal dimension is then reattached to each point in :
to form the set . To encode the distribution of STKs in the 4D space, we consider a 4D regular geometric object called polychoron which is a 4D extension of the 2D polygon. The vertices of a regular polychoron divide the 4D space uniformly, and therefore, each vertex can be considered as a histogram bin. In particular, from the set of regular polychorons, we consider the 120-cell regular polychoron with vertices as given in Table I.
Given the set constructed above, we project each orientation normalized onto the 600 vertices of the polychoron and select the vertex with the highest projection value. The histogram bin corresponding to the selected vertex is incremented by one. We repeat this process for all STKs in and the final histogram is a 600 dimensional STK Distribution (STK-D) descriptor which encodes the global spatio-temporal distribution of STKs of the sequence in a compact and discriminative form.
7 Adaptable Support Volume
So far, for STK detection and description, we have used a fixed spatio-temporal support volume with spatial radius and temporal scale . However, subjects may have different scales (height and width) and may perform actions at different speeds. Therefore, simply using a fixed spatial radius and temporal scale is not optimal. Moreover, a larger value of enables the proposed descriptor to encapsulate more information about shape but makes the descriptor vulnerable to occlusions. Similarly, a small is preferable over large for better temporal action localization. However, a small may not capture sufficient information about an action if it is performed slowly.
7.1 Spatial Scale Selection
Several automatic spatial scale selection methods have been proposed for 3D object retrieval . We adapt the method proposed by Mian et al. in object retrieval for action recognition in 3D pointcloud sequences. Note that in the human action recognition problem, the subject’s height is available in most cases (which is not the case for object retrieval). Where available, we use the subject’s height () to find an appropriate spatial scale. We select the ratios as where is a constant factor. We have empirically selected the value of to maximize the descriptiveness and robustness of our descriptor to occlusions. In all experiments, we use a fixed value of for all actions, views and datasets. In our experiments in Section 8, we observe that this simple approach achieves almost the same accuracy as the automatic spatial scale selection method adapted from. Once we have selected an appropriate spatial scale , then we proceed to select an appropriate temporal scale .
7.2 Automatic Temporal Scale Selection
Most existing action recognition techniques[5, 9, 8, 6, 44, 47] use a fixed temporal scale. We observe that variations in action execution speed cause significant disparity among the descriptors from the same action (Fig. 5). To make our descriptor robust to action speed variations, we propose an automatic temporal scale selection technique.
Let be a sequence of 3D pointclouds. For a given point and a given temporal scale , we can define the spatial support volume and spatio-temporal support volume of . The covariance matrix of the points falling within can be eigen-decomposed to yield the eigenvalues . These steps are similar to those described in Section 3, except that, in this Section, we repeat these steps for each temporal scale , where is a fixed upper threshold. For each value, we calculate:
The optimal temporal scale for the given point is chosen to be the one that minimizes over the range , i.e.,
As an example to illustrate this automatic temporal scale selection process, Fig. 5(a)-(c) show the temporal sequences of pointclouds for the hand waving action performed at three different speeds. The dotted circle shows the sphere defined by the spatial radius in each pointcloud. The spatial radius in the three cases is the same because of similar geometry. Our aim here is to select the optimal temporal scale for the point in the pointcloud shown in black and dotted outline. Figure 5(d) shows the union of points in the range which are within the radius measured from the coordinate () of point . Figure 5(e) and (f) show the union of points in the same way for and , respectively. Figure 5(g)-(i) show the plots of with the variation of . Increasing beyond a certain value does not affect the accumulated pointcloud as the value of becomes constant. In most cases, increasing decreases until a fixed value is reached. We compute for all values of and find the global minimum . When more than one exist, the smallest value of is chosen.
For each STK, the temporal scale is selected independently and may vary from one STK to the other in the same 3D pointcloud sequence. The proposed temporal scale selection is detailed in Algorithm 1. The algorithm outputs two variables and flag . If the optimal is equal to then the flag is set to , indicating that the STK should be discarded. If the computed optimal is smaller than then the flag is set to , indicating that the return value is the optimal temporal scale for .
We evaluate the proposed algorithm on five benchmark datasets, including two multi-view (Northwestern-UCLA Multiview Action3D, and UWA3D Multiview Activity II) and three single-view (MSR Action3D, MSR Daily Activity3D, and MSR Gesture3D) datasets. Performance is compared to nine existing action recognition methods including Histogram of Oriented 4D Normals (HON4D), Super Normal Vector (SNV), Lie Algebra Relative Pairs (LARP), Comparative Coding Descriptor (CCD), Virtual Views (VV), Histogram of 3D Joints (HOJ3D), Discriminative Virtual Views (DVV), Actionlet Ensemble (AE), and AND-OR graph (AOG). The baseline results are obtained using publicly available implementations of CCD, VV, DVV, HON4D, SNV, and LARP from the respective authors’ websites. For the remaining three methods AOG, HOJ3D, and AE, we use our implementations because their codes are not publicly available. For CCD, VV and DVV, we use DSTIP, which is more robust to 3D sensor noise compared to color-based interest point detectors, to extract and describe the spatio-temporal interest points. Our algorithm is robust to many different parameter settings (see Section 8.7). To help the reader reproduce our results, we provide the parameter values that we used in Table II. The UWA3D Multiview Activity II dataset and code will be made publicly available.
To evaluate individual components of the proposed algorithm, we report results for the following four settings:
Holistic HOPC: A sequence of 3D pointclouds is divided into spatio-temporal cells along the , , and dimensions. The spatio-temporal HOPC descriptor in (7) is computed for each point within the sequence. The cell descriptor is computed using (13) and then normalized using (14). The final descriptor for the given sequence is a concatenation of all the cell descriptors. We use SVM for classification. Similar to HON4D and SNV, our Holistic HOPC is suitable for single-view action recognition and can handle more inter-class similarities of local motions compared to local methods.
STK-D: For each sequence of 3D pointclouds, the histogram of spatio-temporal distribution of STKs is used as the sequence descriptor (Section 6.3).
Local HOPC: For each sequence, STKs are detected using the method proposed in Section 4. The proposed orientation normalization is then applied at each STK neighborhood to extract its view-invariant HOPC descriptor (Section 5). The BoW approach is used to describe the sequence.
Local HOPC+STK-D: The bag of STK descriptors and the histogram of spatio-temporal distribution of STKs are concatenated to form the sequence descriptor.
8.1 N-UCLA Multiview Action3D Dataset
The Northwestern-UCLA dataset contains RGB, depth and human skeleton positions captured simultaneously by three Kinect cameras. It consists of 10 action categories: (1) pick up with one hand, (2) pick up with two hands, (3) drop trash, (4) walk around, (5) sit down, (6) stand up, (7) donning, (8) doffing, (9) throw, and (10) carry. Each action was performed by subjects to times. Figure 6 shows sample 3D pointclouds of four actions captured by the three cameras.
To compare our method with state-of-the-art algorithms, we use the same experimental setting as, using the samples from the first two cameras as training data, and the samples from the third camera as test data. Results are given in Table III. Holistic approaches such as HON4D, SNV, and the proposed Holistic HOPC achieved low recognition accuracy since they are not designed to handle viewpoint changes. Similarly, since depth is a function of viewpoint, CCD achieved low accuracy by encoding the differences between the depth values of an interest point and its neighbourhood points.
obtained high accuracy on only RGB videos. On depth videos, AOG also did not perform well. A possible reason is that depth videos have higher noise levels and interpolating noisy features across views can compromise discrimination ability.
Skeleton based methods such as AE and LARP achieved high accuracy. We used the scale and orientation normalization of skeletons proposed in LARP for AE as well which improved the results of AE. However, skeleton data may not be reliable, or even available, when the subject is not in an upright position or is occluded. More importantly, the application of these methods is limited to human activity recognition where the human skeleton is generally estimated by.
STK-D alone achieved higher accuracy compared to all depth image based methods. This confirms the repeatability of STKs and the robustness of STK-D to viewpoint changes. The Local HOPC descriptor achieved higher accuracy than STK-D and all depth based methods. Since HOPC and STK-D capture complementary information, their combination (Local HOPC+STK-D) further improved the performance by % achieving the overall best accuracy of %. Note that this is about % higher than the nearest competitor LARP which requires skeleton data whereas our method does not.
The confusion matrix of our proposed view-invariant Local HOPC+STK-D method is shown in Fig.7. The action (7) donning and action (8) doffing have maximum confusion with each other because the motion and appearance of these actions are very similar. Similarly, action (1) pick up with one hand and action (3) drop trash have high confusion due to similarity in motion and appearance.
8.2 UWA3D Multiview Activity II Dataset
This dataset was collected in our lab using Kinect to emphasize three points: (1) Larger number of human activities. (2) Each subject performed all actions in a continuous manner with no breaks or pauses. Therefore, the start and end positions of body for the same actions are different. (3) Each subject performed the same actions four times while imaged from four different views: front view, left and right side views, and top view.
This dataset consists of human activities performed by subjects with different scales: (1) one hand waving, (2) one hand Punching, (3) two hand waving, (4) two hand punching, (5) sitting down, (6) standing up, (7) vibrating, (8) falling down, (9) holding chest, (10) holding head, (11) holding back, (12) walking, (13) irregular walking, (14) lying down, (15) turning around, (16) drinking, (17) phone answering, (18) bending, (19) jumping jack, (20) running, (21) picking up, (22) putting down, (23) kicking, (24) jumping, (25) dancing, (26) moping floor, (27) sneezing, (28) sitting down (chair), (29) squatting, and (30) coughing. To capture depth videos, each subject performed activities times in a continuous manner. Each time, the Kinect was moved to a different angle to capture the actions from four different views. Note that this approach generates more challenging data than when actions are captured simultaneously from different viewpoints. We organized our dataset by segmenting the continuous sequences of activities. The dataset is challenging because of varying viewpoints, self-occlusion and high similarity among activities. For example, the actions (16) drinking and (17) phone answering have very similar motion, but the location of hand in these two actions is slightly different. Also, some actions such as (10) holding head and (11) holding back, have self-occlusion. Moreover, in the top view, the lower part of the body was not properly captured because of occlusion. Figure 8 shows sample pointclouds of five actions from views.
|AE (Norm. Skeleton)||45.0||40.4||35.1||36.9||34.7||36.0||49.5||29.3||57.1||35.4||49.0||29.3||39.8|
|LARP (Norm. Skeleton)||49.4||42.8||34.6||39.7||38.1||44.8||53.3||33.5||53.6||41.2||56.7||32.6||43.4|
|Holistic HOPC (Depth)||32.3||25.2||27.4||17.0||38.6||38.8||42.9||25.9||36.1||27.0||42.2||28.5||31.8|
|Local HOPC (Depth)||42.3||46.5||39.1||49.8||35.0||39.3||51.9||34.4||57.9||35.3||60.5||36.5||44.0|
|Local HOPC+STK-D (Depth)||52.7||51.8||59.0||57.5||42.8||44.2||58.1||38.4||63.2||43.8||66.3||48.0||52.2|
For cross-view action recognition, we use the samples from two views as training data, and the samples from the two remaining views as test data. Table IV summarizes our results. Since this dataset is more challenging compared to the N-UCLA dataset, the performance of all methods drops significantly. Our Holistic HOPC descriptor achieved higher average recognition accuracy than the depth based methods but lower than the methods which use normalized skeleton data. Among the depth based methods, HON4D and SNV are the nearest competitors to the Holistic HOPC. The Local HOPC achieved higher accuracy than STK-D and the Holistic HOPC. Combining STK-D with Local HOPC again improved performance by % achieving the overall best performance of %. Note that this is about % higher than the nearest competitor LARP which uses skeleton data. Local HOPC+STK-D achieved the highest accuracy in all combinations of training and test views except one. The accuracy of skeleton based methods is significantly lower on this dataset because the skeleton data is not accurate for some actions such as drinking, phone answering, sneezing or is not available for some actions such as falling down and lying down.
Moreover, the overall accuracy of the knowledge transfer based methods VV, DVV, and AOG when depth videos are used as input data is low because motion and appearance of many actions are very similar and the depth sequences have a high level of noise. Therefore, the view dependent local features used in VV, DVV and the appearance and motion interpolation based method used in AOG are not enough to discriminate between actions in the presence of noise.
Figure 9 shows the confusion matrix of our proposed view-invariant Local HOPC+STK-D method when videos from view and view are used for training and videos from view are used as test data. The actions that causes the most confusion are (9) holding chest versus (11) holding back and (12) walking versus (13) irregular walking, because the motion and appearance of these two actions are very similar.
8.3 MSR Action3D Dataset
The MSR Action3D dataset consists of actions performed to times by subjects. This dataset is challenging due to high inter-action similarities. Following the protocol of, we use subjects for training and the remaining for testing and exhaustively repeated the experiments folds. Table V compares our algorithms with existing methods. The proposed Holistic HOPC outperformed all methods and achieved average accuracy which is more than % higher than its nearest competitor SNV and significantly higher than the skeleton based methods such as HOJ3D and LARP. The average accuracy of our view-invariant Local HOPC+STK-D method is which is still higher than HOJ3D, AE , HON4D, and LARP.
8.4 MSR Daily Activity3D Dataset
This dataset contains daily activities performed twice by subjects, once in standing position and once while sitting. Most activities involve human-object interactions which makes this dataset challenging. We follow the experimental setting of and use samples from half of the subjects as training data, and the rest as test data. As shown in Table V, the proposed Holistic HOPC outperformed all techniques achieving an average accuracy of . The view-invariant Local HOPC+STK-D outperformed AOG, HOJ3D and LARP; however, it achieved lower accuracy than HON4D and SNV, because these methods assume that the training and test samples are obtained from the same viewpoint.
8.5 MSR Gesture3D Dataset
The MSR Gesture3D dataset contains American sign language gestures performed to times by subjects. For comparison with previous techniques, we use the leave-one-subject-out cross validation scheme proposed by. Table V compares our methods to existing ones excluding AE, LARP, AOG and HOJ3D since they require 3D joint positions which are not present in this dataset. Our Holistic HOPC outperformed all techniques and achieved an average accuracy of . The Local HOPC+STK-D achieves an accuracy of which is higher than HON4D.
|AE (Norm. Skeleton)||81.6||85.8||NA|
|LARP (Norm. Skeleton)||78.8||69.4||NA|
|Holistic HOPC (Depth)||86.5||88.8||96.2|
|Local HOPC+STK-D (Depth)||82.9||78.8||93.6|
8.6 Effects of Adaptable Support Volume
8.6.1 Spatial Scale Selection
In this experiment, we evaluate the influence of three different approaches for spatial scale selection at each STK. In the first approach, we use a constant spatial scale for all subjects. In the second approach, we select a scale for each subject relative to the subject’s height. In the third one, we use the automatic spatial scale selection method proposed by Mian et al.. Table VI shows the average accuracy of the proposed method in the three settings. Using the subject’s height to find a subject specific scale for the STKs turns out to be the best approach. Automatic scale selection performs closely and can be a good option if the subject’s height cannot be measured due to occlusions. Constant scale for all subjects performs the worst. However, the performance of our algorithm is better than existing techniques in all three settings.
|Dataset||Spatial scale selection method|
8.6.2 Automatic Temporal Scale Selection
We evaluate the improvement gained by our method using automatic temporal scale selection by repeating our experiments with constant temporal scale for STK detection and Local HOPC descriptor extraction. Table VII shows the average recognition accuracy of our proposed method using a constant temporal scale () and automatic temporal scale selection. The proposed automatic temporal scale selection technique achieved higher accuracy which shows the robustness of our method to action speed variations.
|Dataset||Temporal scale selection method|
8.7 Evaluation of Parameters and Computation Time
8.7.1 Number of STKs
To study the effect of the total number of STKs (), we select STKs with the top quality factors as shown in Fig. 10. Note how the STK detector effectively captures the movement of the hands in the highest quality STKs, and noisy points begin to appear as late as . Figure 11(a) shows the influence of the number of STKs on the average recognition accuracy. The proposed method achieves the best recognition accuracy when ; however, the performance remains stable up to .
8.7.2 Threshold Values
We evaluate the effect of the eigenratio thresholds for STK detection in (9), for view-invariant Local HOPC in Section 5, and for STK-D in Section 6.3 on the average recognition accuracy of our proposed method. Figures 11(b)-(d) show our results. Notice that there is a large range () over which the recognition accuracy remains stable. For very small values of , a unique coordinate basis can not be obtained and for larger values of , the number of detected STKs is not sufficient.
A more stable trend in recognition accuracy can be observed for varying the thresholds and . The recognition accuracy starts to decrease when because the number of points within the spatio-temporal support volume of STKs which have unique eigenvectors starts to decrease. Finally, varying does not change the accuracy significantly because the extracted STKs from most actions already have a unique orientation and the proposed iterative refinement process (Section 6.3) almost always finds an unambiguous orientation for all values of .
8.7.3 Computation Time
The average computational time of the STK detection is seconds per frame on a GHz machine with GB RAM using Matlab. The calculation of Local HOPC at STKs takes seconds per frame. The overall computational time of the proposed view-invariant method is about seconds per frame. However, the proposed view-dependent Holistic HOPC is faster and take only seconds per frame. The average computation time of the nearest competitor AOG that uses depth images is seconds per frame. However, our method outperforms AOG on single-view and multi-view datasets by significant margins. Moreover, the calculation of STK and HOPC are individually parallel in nature and can be implemented on a GPU.
9 Discussion and Conclusion
Performance of the current 3D action recognition techniques degrades under viewpoint variations because they treat 3D videos as depth image sequences. Depth images are defined with respect to a particular viewpoint and are thus highly dependent on the viewpoint. We have proposed an algorithm for cross-view action recognition which directly processes 3D pointcloud videos to achieve robustness to variations in viewpoint, subject scale and action speed. We have also proposed the HOPC descriptor that is well integrated with our proposed spatio-temporal keypoint (STK) detection algorithm. Local HOPC descriptor combined with global STK-Distribution achieve state-of-the-art results on two standard cross-view action recognition datasets.
Unlike HOJ3D, LARP, AE, and AOG, our method does not require skeleton data. Skeleton or joint estimation methods such as do not work well when the human body is only partially visible. Moreover, joint estimation may not be reliable when the subject is not in an upright position (e.g. patient lying on bed) or touches the background. Finally, surveillance cameras are usually mounted at elevated angles which causes further difficulties in joint estimation. Thus, our proposed methods (and other non-skeleton based methods) are more general in the sense that they can be applied to a wider variety of action recognition problems where skeletonization of the data is either not possible or has not been achieved yet.
We thank the authors of for providing the Northwestern-UCLA Multiview Action3D dataset and especially Dr Jiang Wang for answering our questions about the implementation of AOG and AE methods. We also thank the authors of[3, 59, 9, 10, 8, 36, 60] for making their codes publicly available. This research is supported by ARC Discovery grant DP110102399.
-  J. K. Aggarwal and M. S. Ryoo, “Human activity analysis: A review,” in ACM Computing Survey, 2011.
-  D. Weinland, R. Ronfard, and E. Boyer, “A survey of vision-based methods for action representation, segmentation and recognition,” in CVIU, 2011.
-  Z. Cheng, L. Qin, Y. Ye, Q. Huang, and Q. Tian, “Human daily action analysis with multi-view and color-depth data,” in ECCVW, 2012.
-  C. Lu, J. Jia, and C. keung Tang, “Range-sample depth feature for action recognition,” in CVPR, 2014.
-  H. Rahmani, A. Mahmood, D. Huynh, and A. Mian, “Action classification with locality-constrained linear coding,” in ICPR, 2014.
-  H. Rahmani, A. Mahmood, A. Mian, and D. Huynh, “Real time action recognition using histograms of depth gradients and random decision forests,” in WACV, 2014.
-  X. Yang, C. Zhang, and Y. Tian, “Recognizing actions using depth motion maps-based histograms of oriented gradients,” in ACM ICM, 2012.
-  L. Xia and J. Aggarwal, “Spatio-temporal depth cuboid similarity feature for activity recongition using depth camera,” in CVPR, 2013.
-  O. Oreifej and Z. Liu, “HON4D: histogram of oriented 4D normals for activity recognition from depth sequences,” in CVPR, 2013.
-  X. Yang and Y. Tian, “Super normal vector for activity recognition using depth sequences,” in CVPR, 2014.
-  H. Zhang, W. Zhou, C. Reardon, and L. E. Parker, “Simplex-based 3D spatio-temporal feature description for action recognition,” in CVPR, 2014.
-  Y. Tian, R. Sukthankar, and M. Shah, “Spatiotemporal deformable part models for action detection,” in CVPR, 2013.
-  M. Zanfir, M. Leordeanu, and C. Sminchisescu, “The Moving Pose: An efficient 3D kinematics descriptor for low-latency action recognition and detection,” in ICCV, 2013.
X. Yang and Y. Tian, “EigenJoints-based action recognition using naive bayes nearest neighbor,” inCVPRW, 2012.
-  A. Yilmaz and M. Shah, “Action sketch: a novel action representation,” in CVPR, 2005.
-  T. Syeda-Mahmood, A. Vasilescu, and S. Sethi, “Action recognition from arbitrary views using 3D exemplars,” in ICCV, 2007.
-  F. Lv and R. Nevatia, “Single view human action recognition using key pose matching and viterbi path searching,” in CVPR, 2007.
-  D. Gavrila and L. Davis, “3D model-based tracking of humans in action: a multi-view approach,” in CVPR, 1996.
-  T. Darrell, I. Essa, and A. Pentland, “Task-specific gesture analysis in real-time using interpolated views,” in PAMI, 1996.
-  B. Li, O. Camps, and M. Sznaier, “Cross-view activity recognition using hankelets,” in CVPR, 2012.
-  V. Parameswaran and R. Chellappa, “View invariance for human action recognition,” in IJCV, 2006.
-  C. Rao, A. Yilmaz, and M. Shah, “View-invariant representation and recognition of actions ,” in IJCV, 2002.
-  T. Syeda-Mahmood, A. Vasilescu, and S. Sethi, “Recognizing action events from multiple viewpoints,” in IEEE Workshop on Detection and Recognition of Events in Video, 2001.
-  S. Seitz and C. Dyer, “View-invariant analysis of cyclic motion,” in IJCV, 1997.
-  M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri, “Actions as space-time shapes,” in ICCV, 2005.
-  D. Weinland, R. Ronfard, and E. Boyer, “Free viewpoint action recognition using motion history volumes,” in CVIU, 2006.
-  L. Xia, C. C. Chen, and J. K. Aggarwal, “View invariant human action recognition using histograms of 3D joints,” in CVPRW, 2012.
-  J. Wang, Z. Liu, Y. Wu, and J. Yuan, “Learning actionlet ensemble for 3D human action recognition,” in PAMI, 2013.
-  J. Wang, X. Nie, Y. Xia, Y. Wu, and S. Zhu, “Cross-view action modeling, learning and recognition,” in CVPR, 2014.
-  A. Gupta, J. Martinez, J. J. Little, and R. J. Woodham, “3D pose from motion for cross-view action recognition via non-linear circulant temporal encoding,” in CVPR, 2014.
-  J. Zheng and Z. Jiang, “Learning view-invariant sparse representations for cross-view action recognition,” in ICCV, 2013.
-  Z. Zhang, C. Wang, B. Xiao, W. Zhou, S. Liu, and C. Shi, “Cross-view action recognition via a continuous virtual path,” in CVPR, 2013.
-  A. Farhadi and M. K. Tabrizi, “Learning to recognize activities from the wrong view point,” in ECCV, 2008.
-  A. Farhadi, M. K. Tabrizi, I. Endres, and D. A. Forsyth, “A latent model of discriminative aspect,” in ICCV, 2009.
-  J. Liu, M. Shah, B. Kuipersy, and S. Savarese, “Cross-view action recognition via view knowledge transfer,” in CVPR, 2011.
-  R. Li and T. Zickler, “Discriminative virtual views for cross-view action recognition,” in CVPR, 2012.
-  H. Rahmani, A. Mahmood, D. Q Huynh, and A. Mian, “HOPC: Histogram of oriented principal components of 3D pointclouds for action recognition,” in ECCV, 2014.
-  W. Li, Z. Zhang, and Z. Liu, “Action recognition based on a bag of 3D points,” in CVPRW, 2010.
-  J. Wang, Z. Liu, Y. Wu, and J. Yuan, “Mining actionlet ensemble for action recognition with depth cameras,” in CVPR, 2012.
-  J. Wang, Z. Liu, J. Chorowski, Z. Chen, and Y. Wu, “Robust 3D action recognition with random occupancy patterns,” in ECCV, 2012.
-  X. Yang and Y. Tian, “A survey of vision-based methods for action representation, segmentation and recognition,” 2011.
-  L. Campbell and A. Bobick, “Recognition of human body motion using phase space constraints,” in ICCV, 1995.
-  J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, “Real-time human pose recognition in parts from single depth images,” in CVPR, 2011.
-  A. W. Vieira, E. Nascimento, G. Oliveira, Z. Liu, and M. Campos, “STOP: space-time occupancy patterns for 3D action recognition from depth map sequences,” in CIARP, 2012.
-  S. Tang, X. Wang, X. Lv, T. Han, J. Keller, Z. He, M. Skubic, and S. Lao, “Histogram of oriented normal vectors for object recognition with a depth sensor,” in ACCV, 2012.
-  I. Laptev, “On space-time interest point,” in IJCV, 2005.
-  P. Dollár, V. Rabaud, G. Cottrell, and S. Belongie, “Behavior recognition via sparse spatio-temporal features,” in ICCV, 2005.
-  G. Willems, T. Tuytelaars, and L. Gool, “An efficient dense and scale-invariant spatio-temporal interest point detector,” in ECCV, 2008.
-  H. Wang and C. Schmid, “Action recognition with improved trajectories,” in ICCV, 2013.
-  H. Wang, A. Klaser, C. Schmid, and C.-L. Liu, “Action recognition by dense trajectories,” in CVPR, 2011.
-  S. Wu, O. Oreifej, and M. Shah, “Action recognition in videos acquired by a moving camera using motion decomposition of Lagrangian particle trajectories,” in ICCV, 2011.
-  N. J. Mitra and A. Nguyen, “Estimating surface normals in noisy point clouds data,” in SCG, 2003.
S. Maji, A. C. Berg, and J. Malik, “Classification using intersection kernel support vector machines is efficient,” inCVPR, 2008.
Y. W. Chen and C. J. Lin, “Combining SVMs with various feature selection strategies,” inFeature Extraction, ser. Studies in Fuzziness and Soft Computing. Springer Berlin Heidelberg, 2006.
-  J. Xie and C. Wang, “Using support vector machines with a novel hybrid feature selection method for diagnosis of erythemato-squamous diseases,” in Expert Systems with Applications, 2011.
-  H. S. M. Coxeter, “Regular polytopes.” Dover Publications, 1973.
-  F. Timbari and L. D. Stefano, “Performance evaluation of 3D keypoint detectors,” in IJCV, 2013.
-  A. Mian, M. Bennamoun, and R. Owens, “On the repeatability and quality of keypoints for local feature-based 3D object retrieval from cluttered scenes,” in IJCV, 2010.
-  R. Vemulapalli, F. Arrate, and R. Chellappa, “Human action recognition by representing 3D skeletons as points in a Lie group,” in CVPR, 2014.
-  R. Gopalan, R. Li, and R. Chellapa, “Domain adaption for object recognition: An unsupervised approach,” in ICCV, 2011.