2D and 3D multi-person pose estimation methods have been intensively researched in computer vision for a few decades. They are needed to give machines a better understanding of human activities and thereby enable the analysis of complex visual content and the development of more sophisticated human-computer interactions. Detecting and localizing parts of an articulated object is a difficult task because it requires to capture both part appearances and the structure of the object. Although the rise of deep learning has lead to large improvements, there are still challenging situations, such as crowded scenes with occluded and interleaved keypoints. Modern operating rooms present such scenarios with high visual complexity due to occlusion and clutter from various equipment, loose clothes worn by clinicians, and the proximity of the persons present in the scene, as illustrated in Fig.1. Our evaluation results reported in  show drastic drops in the performance of state-of-the-art methods on challenging OR data.
The availability of large-scale annotated datasets has been key to the recent progress in human detection and pose estimation. 2D datasets such as Microsoft COCO and MPII include scenes with a wide amount of variability. 3D multi-view datasets, such as Human3.6M, HumanEva, and IXMAS, provide both 2D and 3D annotations, but with a single person performing actions in a controlled environment. As these 3D datasets do not cover real-world challenges, models trained on such data do not generalize well to these complex scenes. The MultiHumanOR dataset introduced in  is a multi-view OR dataset with 2D and 3D human poses. However, it was captured during activities simulated by actors.
We wish to introduce a new multi-view dataset illustrating the complexity of the challenging OR environment and containing data captured during real interventions. It is the first public dataset for human pose estimation featuring real clinical data and also the first multi-person multi-view RGB-D dataset containing 3D poses captured during real activities. We hope that releasing this dataset will foster research in 2D and 3D human pose estimation and the development of the related clinical applications.
We present below the dataset, the annotations and the results of several baselines. Since some parts of the images needed to be blurred for anonymization, we also present a comparative study illustrating the impact of the blurring process on the results.
2 MVOR Dataset
The MVOR dataset consists of 732 multi-view frames111We define a multi-view frame as the set of RGB-D images recorded from all cameras at the same time step. sampled from four days of recording in an interventional room at the University Hospital of Strasbourg during procedures such as vertebroplasty and lung biopsy. The ground truth annotations consist of 4699 human bounding boxes, 2926 2D upper-body poses and 1061 3D upper-body poses. The multi-view recording was performed using three RGB-D cameras (Asus Xtion Pro) mounted on the ceiling using articulated arms. The cameras were mounted in such a way as to capture the key activities around the operating table as shown in Fig. 2. Multi-view person and part visibility statistics are shown for the 3D annotations in Table 1. The image and depth data were captured at 20 FPS in 640x480 VGA resolution using a recording software developed in-house. The intrinsic camera parameters of each camera were computed using a calibration pattern. The rigid transformation between the cameras and transformation of each camera to the global coordinate system were done in the two-step process as described in [14, 10]. The operating table was considered to be the reference for the global coordinate system.
In the dataset, the color images needed to be blurred to ensure the anonymization of the data. We have tried to minimize the amount of blurring so that computer vision algorithms would not be impacted. Patient faces and nude parts are fully blurred, while clinicians’ faces are only blurred around the eyes when wearing a mask, and fully blurred otherwise. A sample image is shown in Fig. 3 (left).
This dataset contains annotations for clinical staff and patients. All persons are annotated with a full bounding box and staff who have more than 50% of their upper-body parts visible in at least one view are annotated with 2D and 3D upper-body pose keypoints. The 10 keypoints annotating the upper-body poses are shown in Fig. 3 (right).
To generate the annotations, we use a tool that displays all the three 2D views as well as the 3D point cloud, illustrated in Fig. 4. First, the annotator draws the 2D skeletons in all 2D views. To generate the 3D annotations, the 2D poses are back-projected into 3D using the depth information and initial 3D skeletons are computed by averaging all 3D skeletons across all views. We compute average 3D joint locations only among visible body joints. These initial 3D skeletons are not always accurate due to depth errors and differences in 2D joint annotations among the views, which are in turn caused by the large visual differences due to cameras rotation angles and partial occlusions. The annotator is therefore required to then ensure the quality of each 3D skeleton by verifying the accuracy of its projections to all views and by updating its locations directly in 3D when needed. Examples of available 2D/3D annotations are shown in Figure 5.
3 Baseline methods
We briefly present the methods that we have evaluated on the MVOR dataset for 2D/3D pose estimation and for person detection.
3.1 2D pose estimation
This is a part-based approach which relies on both color and depth images to perform human pose estimation. First, a deep convolutional neural network is used to detect body parts. Then, part detection score maps and depth information are used to perform exact and efficient inference in 3D. This approach is finetuned on another RGB-D dataset captured in the same OR.
OpenPose: This is an open-source222The public implementation used in the evaluation is available at https://github.com/CMU-Perceptual-Computing-Lab/openpose bottom-up method, particularly well suited for real-time detections in RGB images. A deep multi-stage and two-branch CNN jointly predict heatmaps and part affinity fields to capture body parts and pairwise dependencies between body joints. Keypoints are then assembled into skeletons through a bipartite graph matching algorithm. For the sake of comparison, we test both OpenPose with and without multi-scale testing. We also compare to the RTPose implementation666The public implementation used in the evaluation is available at https://github.com/CMU-Perceptual-Computing-Lab/caffe_rtpose, the previous software version of OpenPose, which shows better results.
AlphaPose: This is an open-source333The public implementation used in the evaluation is available at https://github.com/MVIG-SJTU/AlphaPose top-down method for 2D pose estimation in RGB images. It performs human detection with Faster-RCNN and single person pose estimation on the extracted bounding boxes.
3.2 3D pose estimation
MV3DReg: This approach formulates the problem of human pose estimation from multiple RGB views in a two-step framework. The method first detects 2D poses in each view independently and then relies on a neural network to regress for 3D poses. In order to easily generalize to a new environment, the approach models the characteristics of the 2D detector used at test time and proposes to only rely on a set of plausible 3D poses to learn the 3D regression function.
3.3 Person detection
We evaluate person detection from RGB images with state-of-the-art convolutional object detectors, using only the detections corresponding to the person category, and with 2D pose estimators by extracting bounding boxes from predicted skeletons.
. It enhances the R-CNN framework by making the region proposal network fully convolutional. Deep features are used instead of the input image to select the region of interests with a sliding window approach. Then, a second network classifies and refines the bounding box for each region of interest.
Deformable convolutional networks on R-fcn: This approach555The public implementation used in the evaluation is available at https://github.com/msracver/Deformable-ConvNets introduces a new kind of convolutional layer that extends the sampling grid of each kernel by learning an offset for each weight. These layers are integrated into the R-fcn architecture, a fully convolutional two-stage object detector.
In this section, we evaluate the state of the art methods described in the previous section for person detection and 2D/3D human pose estimation on the MVOR dataset. We also show a comparative study illustrating how these methods are affected by the blurring process. In the tables, we refer to the original (i.e, non-blurred) images with O and to the blurred images with B.
4.1 2D pose estimation
We use the percentage of correct keypoints (PCK) to compare the baseline pose estimation methods. This metric measures the localization accuracy of the body joint, based on the scale of the person. To match detected and ground-truth skeletons, a tight bounding box is computed for each ground-truth skeleton from its keypoints. Then, for each ground-truth skeleton, we select the detection with the highest confidence score among the detections which have more than 30% of their keypoints in the ground-truth bounding-box.
Table 2 shows the results for AlphaPose, Deep3DPS and three versions of OpenPose. None of the 2d pose estimation algorithm is affected by the blurring process.
4.2 3D pose estimation
The evaluation results of MV3DReg are presented in Table 3 using the 3D mean per joint position error (MPJPE) in centimeters. We present the results per number of supporting views. The approach performs similarly on both blurred and non-blurred images, which is on par with 2D detection results222Note that MV3DReg is using 2D detections from the OpenPose implementation.. This confirms the hypothesis that the blurring process has a negligible effect on the performance of pose estimation approaches. These results also indicate that the localization error decreases as the number of supporting view increases. This demonstrates the great benefit of multi-view data for localizing body parts in cluttered environments.
4.3 Person detection
For person detection from object detectors, we use the standard metrics from COCO, i.e average precision (AP), along with average recall () for a fixed number of detections. is the average precision for a fixed intersection over union (IoU). AP is the average of for IoU from 0.50 to 0.95, with a step size of 0.05. (with ) is the average of for IoU from 0.50 to 0.95, with a step size of 0.05.
For person detection from pose estimation, we choose to only report , because the extracted boxes are inherently smaller than the ground-truth and cannot intersect them completely, which leads to low for . In the same manner, we only compute AR for an IoU of .
Again, the difference between the results on blurred and original images is very little. As expected, the average recall is always greater on original images. However, there is sometimes a better average precision on blurred images, since there are fewer detections due to blurring.
In this paper, we present and release a new multi-view dataset for multi-person detection and 2D/3D human pose estimation in a challenging environment, namely a modern operating room, which contains inherent visual challenges such as multiple occlusions. We also present the results of several recent baseline methods as well as a comparative study showing that the blurring process required for medical confidentiality only affects mildly the accuracy of detections. This dataset can thus be helpful to evaluate a detector’s ability to generalize to unseen configurations and color distribution and also to assess the performance of 3D multi-person pose estimation methods on real-world data.
This work was supported by French state funds managed by the ANR within the Investissements d’Avenir program under references ANR-16-CE33-0009 (DeepSurg), ANR-11-LABX-0004 (Labex CAMI) and ANR-10-IDEX-0002-02 (IdEx Unistra). The authors would also like to thank the members of the Interventional Radiology Department at University Hospital of Strasbourg for their help in generating the dataset.
Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New benchmark and state of the art analysis. In: Proceedings of the IEEE Conference on computer Vision and Pattern Recognition. pp. 3686–3693 (2014)
-  Belagiannis, V., Wang, X., Shitrit, H.B.B., Hashimoto, K., Stauder, R., Aoki, Y., Kranzfelder, M., Schneider, A., Fua, P., Ilic, S., Feussner, H., Navab, N.: Parsing human skeletons in an operating room. Mach. Vis. Appl. 27(7), 1035–1046 (Oct 2016)
-  Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multiperson 2d pose estimation using part affinity fields. CVPR (2017)
-  Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 764–773 (Oct 2017)
-  Dai, J., Li, Y., He, K., Sun, J.: R-fcn: Object detection via region-based fully convolutional networks. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 379–387 (2016)
-  Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1325–1339 (Jul 2014)
-  Kadkhodamohammadi, A., Gangi, A., de Mathelin, M., Padoy, N.: A multi-view RGB-D approach for human pose estimation in operating rooms. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 363–372 (Mar 2017)
-  Kadkhodamohammadi, A., Padoy, N.: A generalizable approach for multi-view 3D human pose regression. CoRR abs/1804.10462 (2018)
-  Lin, T.Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Lawrence Zitnick, C., Dollár, P.: Microsoft COCO: Common objects in context. CoRR abs/1405.0312 (2014)
-  Loy Rodas, N., Padoy, N.: Seeing is believing: increasing intraoperative awareness to scattered radiation in interventional procedures by combining augmented reality, monte carlo simulations and wireless dosimeters. Int. J. Comput. Assist. Radiol. Surg. 10(8), 1181–1191 (Aug 2015)
-  Lu, C.: Rmpe : Regional multi-person pose estimation. ICCV (2017)
-  Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards Real-Time object detection with region proposal networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28, pp. 91–99 (2015)
-  Sigal, L., Balan, A.O., Black, M.J.: HumanEva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. Int. J. Comput. Vis. 87(1), 4 (Aug 2009)
-  Svoboda, T., Martinec, D., Pajdla, T.: A convenient multicamera Self-Calibration for virtual environments. Presence: Teleoperators and Virtual Environments 14(4), 407–422 (Aug 2005)
-  Weinland, D., Özuysal, M., Fua, P.: Making action recognition robust to occlusions and viewpoint changes. In: Computer Vision – ECCV 2010. pp. 635–648. Springer Berlin Heidelberg (2010)
-  Yang, Y., Ramanan, D.: Articulated human detection with flexible mixtures of parts. IEEE transactions on pattern analysis and machine intelligence 35(12), 2878–2890 (2013)