Hybrid calibration procedure for fringe projection profilometry based on stereo-vision and polynomial fitting

03/09/2020 ∙ by Raul Vargas, et al. ∙ 0

The key to accurate 3D shape measurement in Fringe Projection Profilometry (FPP) is the proper calibration of the measurement system. Current calibration techniques rely on phase-coordinate mapping (PCM) or back-projection stereo-vision (SV) methods. PCM methods are cumbersome to implement as they require precise positioning of the calibration target relative to the FPP system but produce highly accurate measurements within the calibration volume. SV methods generally do not achieve the same accuracy level. However, the calibration is more flexible in that the calibration target can be arbitrarily positioned. In this work, we propose a hybrid calibration method that leverages the SV calibration approach using a PCM method to achieve higher accuracy. The method has the flexibility of SV methods, is robust to lens distortions, and has a simple relation between the recovered phase and the metric coordinates. Experimental results show that the proposed Hybrid method outperforms the SV method in terms of accuracy and reconstruction time due to its low computational complexity.



There are no comments yet.


page 1

page 2

page 3

page 4

page 5

page 6

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Accurate 3D shape measurement with Fringe Projection Profilometry (FPP) requires proper calibration [Zhang:2016tq, Zhang:2018jb, marrugo2018fourier]. Currently, most calibration procedures in FPP rely on phase-coordinate mapping (PCM) or stereo vision (SV) methods [cai2017phase, Li:2016hk, vargas2018camera]. PCM techniques relate experimental metric coordinates to phase values using polynomial [vo2010flexible, huang2014new] or rational [huang2010least, zhou2014analysis, du2007three] fitting functions. However, they require the use of 2D or 3D targets with well-known dimensions, and positioning them at distances and orientations with high precision [Villa:2012jk]. Though accurate, these requirements lead to expensive calibration experimental setups [Zhang:2016tq].

Alternatively, the extensively adopted SV method provides much more flexibility by calibrating the projector in a similar manner as a camera [zhang2006novel]. Hence, the FPP system can be modeled using the well-established camera calibration methods [hartley2003multiple]. The calibration process can be carried out using arbitrarily placed 2D targets, resulting in a more flexible calibration procedure [zhang2000flexible]. However, it is known that lens distortions introduce significant errors in the SV model 3D reconstruction [vargas2018camera, Vargas:2018vk]. Several strategies have been proposed to compensate for the projector and camera lens distortions to improve accuracy. Some consist of projecting additional patterns to obtain a direct camera-projector pixel correspondence [li2008accurate, moreno2012simple]. Although, they increase the acquisition time by requiring a large number of images or require costly correspondence search algorithms. Other methods consist of digitally pre-deforming the fringe pattern to obtain a distortion-free projected pattern [li2016lens, gonzalez2019accurate]. Nevertheless, the calibration procedure can become elaborate, which decreases the flexibility of the calibration procedure.

In recent years, a different approach has emerged. Mainly, to combine PCM and SV models to obtain a flexible calibration procedure and a simpler calibration model [Chen:2016bi]. For instance, the SV calibration parameters were used to fit a polynomial PCM model, and such PCM model achieved the same accuracy as the SV model, but with higher computational efficiency [cai2017phase, cai2018ray]. Therefore, the question becomes, is it possible to use a PCM model to achieve higher accuracy?

To address the above question, we propose a four-stage hybrid calibration procedure. 1) we carry out a conventional SV calibration of the FPP system considering camera and projector lens distortions; 2) we obtain the pose of a flat board using the SV model 3) we compute pixel-wise error maps to ideal planes, 4) we obtain new pixel-wise relationship between (, , and ) and phase values through pixel-wise polynomial fitting. We will demonstrate that our approach improves the overall accuracy while maintaining a low computational complexity.

Figure 1: (a) Experimental Setup. (b) Stereo model diagram of a camera-projector system.

2 Method and Materials

2.1 Experimental setup

In Fig. 1(a), we show the experimental setup, which consists of a monochromatic CMOS camera Basler Ace-1600gm with resolution of 1600x1200 at 60 fps with a focal length of 12 mm at F1.8 (Edmund Optics-58001), a DLP projector DELL M115HD with native resolution of 1280x800, a checkerboard for calibration with  mm squares, and a PC workstation. To synchronize the camera and DLP projector, we duplicate the VGA signal from the computer using a VGA splitter and connecting the vertical sync pulse to the hardware trigger pin of the camera.

2.2 Stereo vision calibration

In this work, we begin from the stereo calibration method proposed by Zhang and Huang [zhang2006novel] to develop a flexible and robust system calibration approach. The camera-projector system is considered as a binocular framework by regarding the projector as an inverse camera. In Fig. 1(b), we show a schematic of the fringe projection system. Considering a point in the world coordinate system (WCS), we find its corresponding coordinate in the camera or projector systems using the following equation,


where is the scaling factor; and denote the effective focal lengths along and directions, respectively; is the coordinate of the principal point;

is the skew factor of images axes. The matrix

represents a rigid transformation from the world coordinate system (WCS) to the camera coordinate system (CCS) or the projector coordinate system (PCS). Usually, the WCS matches with the CCS and the transformation WCS to PCS is defined as,


where R is a rotation 33 matrix, is a 3

1 translation vector, and

denotes a 31 vector with the Euler angles.

The camera and projector distortion lens are modeled as [vo2011advanced]




where and are the radial and tangential distortion coefficients, respectively; and refer to normalized coordinate before and after the distortion correction.

Figure 2: The FPP system is initially calibrated using the conventional (a) Stereo calibration procedure with a B&W checkerboard. It requires capturing (b)-(f) vertical and horizontal fringe patterns, along with centerline images for absolute phase retrieval, and a texture image.

To obtain the stereo parameters that describe our system, we use a standard stereo calibration procedure using a black and white (B&W) checkerboard. We placed the board at 15 different distances and orientations from the system, as shown in Fig 2(a). In each position, we captured the images shown in Fig. 2(b)-(f). Using the vertical (Fig. 2(b)) and horizontal (Fig. 2(c)) fringe images, we recovered the discontinuous phase employing the phase-shifting algorithm with 8 images. Afterward, we apply a phase unwrapping algorithm using a centerline image (Fig. 2(d) and Fig. 2(e)) to obtain the absolute phase maps in the horizontal () and vertical () directions[zhang2006novel]. Additionally, we capture a texture image of the checkerboard projecting white light on it (Fig. 2(f)).

We calculated the corners with subpixel precision using texture images, and corner coordinates in the projector image plane

were calculated using phase values in each detected corner. To avoid phase errors in the checkerboard corners, we used only the phase from the white squares (having morphologically eroded boundaries to avoid black-to-white transition), and we interpolate the phase values using a 5th order 2D polynomial function to all pixels following a similar procedure as described in Ref. 


Having determined the coordinates and , we used the camera calibration toolbox proposed by Bouguet [bouguet2008camera] to obtain the intrinsic and extrinsic stereo parameters. The obtained reprojection errors for the camera and the projector are 0.154 and 0.099, respectively, which are quite small. In Table 1, we show the stereo parameters obtained for our system.

Parameter Camera Projector
[pix] 794.59, 594.32 632.42, 799.67
[pix] 2698.42, 2701.55 1949.18, 1953.15
0.84 0.83
0.0011 0.0004
0.0006 0.0002
-0.12 0.002
-0.52 -0.077
5.08 -0.033
[] -0.42, 19.08, 0.36
[mm] -163.71, -116.73, 135.41
Table 1: System stereo parameters of SV calibration.
Figure 3: The hybrid calibration procedure, and the resulting 3D reconstruction method.

2.3 Hybrid calibration

In Fig. 3, we show a block diagram of the proposed hybrid calibration procedure. We begin from the standard SV calibration approach. We then use it to obtain the pose of a flat white board at different positions. Suppose that any point () of the 3D reconstruction of each pose , will have small residual errors 

relative to the true 3D surface. We assume that the best guess to the true surface is an ideal plane estimated via least-squares. Accordingly, the residual errors are modeled as the perpendicular distances to that plane. For each reconstructed 3D point along the line-of-sight of a given camera pixel

we calculate the errors relative to each ideal plane. This correction does not generally result in a straight line, but in a smooth curve in space. Finally, we use a pixel-wise third-order polynomial regression model to relate the absolute phase values to the corrected () metric coordinates.

2.3.1 Flat board pose estimation

The mapping procedure that we propose consists of positioning a flat white board within a predefined calibration volume and move it through that volume, as shown in Fig. 4(a). We propose to use a white board since it allows us to have a phase map without the presence of errors caused by intensity variations. Although the plane can be placed arbitrarily throughout the calibration volume, we decided to manually move the plane in 30 positions using homogeneous displacements. This operation guarantees a correct sampling of the region of interest since fitting a third-order polynomial requires a minimum of four experimental phase-coordinate relations. In each position , the planes are reconstructed using the SV model, obtaining the respective , , and coordinates for each camera pixel .

2.3.2 Pixel-wise error function

Nevertheless, the SV-reconstructed coordinates have residual errors. To reduce these errors, we fit each reconstructed plane to an ideal plane by least-squares. Then, each , , and coordinate of the reconstructed plane is corrected as follows,


where , , and represent the corrected coordinates for a camera pixel (), are the estimated residual errors from the perpendicular distance between the ideal plane and each point (, , ). In Fig. 4(b), we show the phase map using vertical fringes and the corrected coordinates for the plane at position .

Figure 4: The (a) Hybrid calibration procedure consists in manually displacing a flat white board along a predefined calibration volume. The (b) recovered absolute phase map and metric coordinates () of an ideal plane estimated for a position of the flat board are used for the input for the hybrid calibration.

2.3.3 Pixel-wise polynomial fit

We use a pixel-wise regression model to relate the absolute phase values to the metric coordinates of the FPP system for each pixel ,


where is the absolute phase value for a camera pixel (); , , and are the corrected metric coordinates in the camera coordinate system; and , and are the coefficients of the adjustment polynomials , and , respectively. We have omitted the pixels in the notation to avoid making the equations more complicated.

Figures 5(a), (b) and (c) show the experimental data and the adjustment polynomial between the phase values and the , and coordinates for the pixel of the camera. Note that the third-order polynomial allows obtaining a correct representation of the data. We calculate an RMS error between the , and corrected experimental data and the fitted polynomials for each pixel for all poses of the white flat board. In Figures 5(d)-(f) we show the RMS errors corresponding to , , and , respectively. The RMS errors of the polynomial fittings have a maximum of 0.060 mm for the coordinate, while for the - and - coordinates, the RMS errors are less than 0.015 mm. Therefore, the third-order polynomial regression model correctly describes our system within the calibrated volume, which is approximately 270 mm 200 mm 130 mm.

2.3.4 3D reconstruction

As shown in Fig. 3, the 3D reconstruction under the proposed hybrid calibration procedure consists of two stages. First, recovering the absolute phase per camera pixel from the projected vertical fringe patterns. Second, evaluating the polynomials from equations (8)-(10) to obtain the accuracy-improved 3D reconstruction.

Figure 5: A third-degree polynomial function describes sufficiently well the phase to ideal (a) , (b) and (c) coordinates of the image pixel (200, 200). The RMS error maps of (d) , (e)  and (f) -calibration show a small error throughout most of the field of view.

3 Experimental Results

3.1 Experiment 1: Reconstruction of a flat board

Here, we compare the accuracy of the SV and Hybrid models. To do this, we reconstructed a flat white board of 230 mm  175 mm positioned within the calibration volume of the hybrid model, and we inclined it relative to the axis of the camera, as shown in Fig. 6(a). This board was reconstructed using the two calibration models, and an adjustment to an ideal plane was carried out by least-squares fitting. In Fig. 6(b), we show the histograms of the plane adjustment error corresponding to each calibration model. Note that the hybrid calibration model improves the accuracy of the reconstruction of the plane, reducing the dispersion of the error distribution and obtaining an RMS error of 0.044 mm, which is lower than the RMS error of 0.118 mm obtained with the SV model. Therefore, although the hybrid model is derived from the SV model, it provides greater precision in the reconstruction by compensating the residual errors. In the figures 6(c) and 6(d), we show the error maps obtained with the SV and the hybrid model, respectively. In Fig. 6(c), we can see that the residual errors in the SV model vary through the field of view in a range of  mm, while the hybrid model has a more homogeneous error distribution with most errors in the range of  mm.

Figure 6: Experiment 1 results. (a) Flat board. (b) Error histograms in the reconstruction of a flat board using SV, and hybrid calibration models. (c) and (d) adjustment error maps of a plane reconstructed by SV and hybrid model, respectively. The hybrid model significantly improves the accuracy of the 3D reconstruction.

3.2 Experiment 2: Reconstruction of a cylinder

In this experiment, we evaluate the performance of the proposed calibration model to perform the reconstruction of a non-flat object, a cylinder. The tested object is a 168mm diameter and 215mm height Polyvinyl chloride (PVC) pipe. A section of the pipe was reconstructed using the SV and Hybrid models. Fig. 7(a) shows the 3D shape of the cylindrical cap reconstructed using the Hybrid model. To assess the shape measurement capability of each method, we adjust each reconstruction to an ideal cylinder by least-squares fitting. The histograms of adjustment errors obtained for each reconstruction are shown in Fig 7(b). Note that the hybrid model allows obtaining a correct cylindrical representation of the reconstruction since the errors are more concentrated around zero, while the histogram of the SV model has a higher dispersion. The RMS adjustment errors obtained with the SV and hybrid models were 0.076 and 0.059 mm, respectively. Figures 7

(c) and (d) show the adjustment error maps for the SV and hybrid models, respectively. Note that in the SV model, the largest residual errors are located in the upper and lower part of the cylinder; meanwhile, in the error map of the hybrid reconstruction, the errors are uniformly distributed throughout the image.

Figure 7: Experiment 2 results. (a) 3D reconstruction shape of a cylinder using the hybrid model. (b) Histogram of Adjustment error of a cylinder reconstructed using the hybrid and SV models. (c) and (d) Error maps of the cylinder reconstructed by SV model and hybrid model, respectively. The accuracy improvement is also displayed in the 3D reconstruction of a non-flat object.

3.3 Experiment 3: Complex surface reconstruction

In this experiment, we performed the reconstruction of a complex ceramic mask of an approximate size of 160mm190mm. To do this, we projected vertical and horizontal fringe patterns onto the surface of the object, as shown in Fig. 8(a). It was necessary to project the horizontal strip patterns to compensate the distortions of the projector in the stereo model, while the hybrid model only requires the projection of vertical fringes. In Fig. 8(b), we show the reconstruction of the ceramic mask using the hybrid model. Fig. 8(c) shows the superposition of the reconstructions of the hybrid and SV models. The RMS distance between the two reconstructions is 0.075 mm. The reconstruction times of the object (683047 points) were 1.04s and 0.14s for the SV models and the hybrid model, respectively.

Figure 8: Experiment 3 results. (a) Captured fringe images of a ceramic mask. (b) 3D reconstruction of the mask. (c) Superposition of the 3D reconstructions using the hybrid and SV models.

3.4 Experiment 4: Execution time assessment

Figure 9: Experiment 4 results. (a) Reconstructed NxN size windows of an object for reconstruction time analysis. (b) reconstruction time curves of the SV and hybrid models using square reconstruction windows with NxN pixels.

Finally, we analyzed the execution times required for the hybrid model and the SV model to reconstruct objects of different sizes. To simulate this, we reconstructed several sections of a study object with different numbers of pixels. The object of reconstruction is the flat board used in the calibration process, and the reconstruction sections were chosen using square windows with a size of pixels, as shown in Fig. 9(a).

In Fig. 9(b), we show the time required to perform the reconstructions of eight different window sizes using the Hybrid and SV models. Note that the hybrid model performs the 3D reconstructions with low computation time compared to the stereo model. This difference is because the SV model demands significant time in the compensation of distortions of the projector, which is carried out with an iterative process. Whereas in the hybrid model, these distortions are implicitly compensated in the polynomial model coefficients for each pixel, and no iterative computation is required.

4 Conclusion

Improving the measurement accuracy of a standard fringe projection profilometry system typically requires elaborate calibration procedures or sophisticated iterative methods to reduce 3D reconstruction error. Here, we have proposed a calibration method that leverages the stereo calibration approach for fringe projection profilometry and improves the measurement accuracy. The hybrid method uses the reconstructions of a flat white board as input to obtain pixel-wise polynomials for converting phase to metric coordinates with higher accuracy than the underlying stereo model. The accuracy improvement was shown through several experiments. Moreover, the low computational complexity of the proposed method reduces the execution time for 3D reconstruction significantly.


Colciencias (project 538871552485), and Universidad Tecnológica de Bolivar (projects C2018P005 and C2018P018).


The authors declare no conflicts of interest.


R. Vargas thanks Universidad Tecnológica de Bolívar (UTB) for a post-graduate scholarship. L.A. Romero and A.G. Marrugo thank UTB for a Research Leave Fellowship. A.G. Marrugo acknowledges support from the Fulbright Commission in Colombia and the Colombian Ministry of Education within the framework of the Fulbright Visiting Scholar Program, Cohort 2019-2020. Parts of this work were presented at the 2019 Iberoamerican Optics Meeting (RIAO) in Cancun, Mexico, and at the 2019 SPIE Optical Metrology conference in Munich, Germany [vargas2019flexible].