Digitally scanning paper documents for sharing and editing is becoming a common daily task. Such paper sheets are often curved or folded, and proper rectification is important for high-fidelity digitization and text recognition. Flatbed scanners allow physical rectification of such documents but are not suitable for hardcover books. For a wider applicability of document scanning, it is wanted a flexible technique for digitally rectifying folded documents.
There are two major challenges in document image rectification. First, for a proper rectification, the 3D shape of curved and folded paper sheets must be estimated. Second, the estimated surface must be flattened without introducing distortions. Prior methods for 3D reconstruction of curved paper sheets either use specialized hardware[1, 2, 3] or assume simplified parametric shapes [4, 5, 6, 7, 8, 9, 2], such as generalized cylinders (Fig. 2a). However, these methods are difficult to use due to bulky hardware or make restrictive assumptions about the deformations of the paper sheet.
In this paper, we present a convenient method for digitally rectifying heavily curved and folded paper sheets from a few uncalibrated images captured with a hand-held camera from multiple viewpoint. Our method uses structure from motion (SfM) to recover an initial sparse 3D point cloud from the uncalibrated images. To accurately recover the dense 3D shape of paper sheet without losing high-frequency structures such as folds and creases, we develop a ridge-aware
surface reconstruction method. Furthermore, to achieve robustness to outliers present in the sparse SfM 3D point cloud caused by repetitive document textures, we pose the surface reconstruction task as a robust Poisson surface reconstruction based onoptimization. Next, to unwrap the reconstructed surface, we propose a robust conformal mapping method by incorporating ridge-awareness priors and optimization technique. See Fig. 1 for an overview.
The contributions of our work are threefold. First, we show how ridge-aware regularization can be used for both 3D surface reconstruction and flattening (conformal mapping) to improve accuracy. Our ridge-aware reconstruction method preserves the sharp structure of folds and creases. Ridge-awareness priors act as non-local regularizers that reduce global distortions during the surface flattening step. Second, we extend the Poisson surface reconstruction  and least-squares conformal mapping (LSCM)  algorithms by explicitly dealing with outliers using optimization. Finally, we describe a practical system for rectifying curved and folded documents that can be used with ordinary digital cameras.
2 Related Work
The topic of digital rectification of curved and folded documents has been actively studied in both the computer vision and document processing communities. It is common to model paper sheets as developable surfaces which have underlying rulers corresponding to lines with zero Gaussian curvature. Many existing methods assume generalized cylindrical surfaces where the paper is curved only in one direction and thus can be parameterized using a 1D smooth function. Such surfaces do not require an explicit parameterization of the rulers. See Fig.2a for an example. A variety of existing techniques recover surface geometry using this assumption. Shape from shading methods were first used by Wada et al. , Tan et al. [12, 13], Courteille et al.  and Zhang et al.  whereas shape from boundary methods were explored by Tsoi et al. [15, 6]. Binocular stereo matching with calibrated cameras was used by Yamashita et al. , Koo et al.  and Tsoi et al. . Shape from text lines is another popular method for reconstructing the document surface geometry [17, 12, 18, 19, 20, 21, 8, 22, 9, 23, 24, 25]. However, these methods assume that the document contains well-formatted printed characters.
Some recent methods relax the parallel ruler assumption (see Fig. 2b). However, the numerous parameters in these models makes the optimization quite challenging. Liang et al.  and Tian et al.  use text lines. Although these methods can handle a single input image, the strong assumptions on surface geometry, contents and illumination limit the applicability. Meng et al.designed a special calibrated active structural light device to retrieve the two parallel 1D curvatures , the surface can be parameterized by assuming appropriate boundary conditions and constraints on ruler orientations. Perriollat et al. use sparse SfM points but assume they are reasonably dense and well distributed. Their parameterization is sensitive to noise and can be unreliable when the 3d point cloud is sparse or has varying density.
For rectification of documents with arbitrary distortion and content (Fig. 2c), other methods require specialized devices and use non-parametric approaches. Brown et al.  use a calibrated mirror system to obtain 3D geometry using multi-view stereo. They unwrap the reconstructed surface using constraints on elastic energy, gravity and collision. The model is not ideal for paper documents because developable surfaces are not elastic. Later, they propose using dense 3D range data  after which they flatten the surface using least square conformal mapping . Zhang et al.  also use dense range scans and use rigid constraints instead of elastic constraints with the method proposed in . Pilu  assumes that a dense 3D mesh is available and minimizes the global bending potential energy to flatten the surface. None of these existing methods are as practical and convenient as our method that only requires a hand-held camera and a few images.
3 Proposed Method
Our method has two main steps – 3D document surface reconstruction and unwrapping of the reconstructed surface. For now, we assume that a set of sparse 3D points on the surface are available. Next, we describe our new algorithms for ridge-aware surface reconstruction and robust surface unwrapping.
3.1 Ridge-aware surface reconstruction
Dense methods are favored for 3D scanning of folded and curved documents [31, 32]. This is because existing methods for surface reconstruction from sparse 3D points tend to produce excessive smoothing and fail to preserve sharp creases and folds ie. ridges on the surface. Such methods are typically also inadequate for dealing with noisy 3D points caused by repetitive textures present in documents. We address these issues by developing a robust ridge-aware surface reconstruction method for sparse 3D points. Specifically, we extend the Poisson surface reconstruction method  by incorporating ridge constraints and by adding robustness to outliers.
Robust Poisson surface reconstruction. We denote a set of sparse 3D points obtained from SfM as , where only 3D points triangulated from at least three images are retained. For our input images, typically lies between to . For a selected reference image (and viewpoint), we use a depth map parameterization for the document surface. We aim to estimate depth at the mesh grid vertices , where is the mesh grid index, . Our method computes the optimal depth values by solving the following optimization problem.
Here, and are the data and smoothness terms respectively and is a parameter to balance the two terms. The original Poisson surface reconstruction method uses the squared -norm for both terms. Instead, we propose using the -norm for the data term to deal with outliers.
This encourages to be consistent with the observed depth without requiring explicit knowledge about which observations are outliers. We rewrite Eq. (2
) in vector form.
where is a permutation matrix that selects and aligns observed entries by ensuring correspondence between and . The smoothness term is defined using the squared Frobenius norm of the gradient of depth vector along and in camera coordinates.
By preparing a sparse derivative matrix that replaces the Laplace operator in a linear form,
we have a special form of the Lasso problem .
While this problem (Eq. (6)) does not have a closed form solution, we use a variant of iteratively reweighted least squares (IRLS)  for deriving the solution. By rewriting the data terms in Eq. (6) as a weighted norm using a diagonal matrix with positive values on the diagonal, we have
Step 1: Update
Eq. (7) can be rewritten as , where and and is a zero vector of length . This is a squared sparse linear system. It has the closed form solution
is the identity matrix,is a regularization parameter (we use ).
Step 2: Update
We initialize to the identity matrix. During each iteration, each diagonal element of is updated given the residual , as follows.
Here, is the -th element of and is a small positive value (we use ). These steps are repeated until convergence; namely, until the estimate at -th iteration becomes similar to the previous estimate , i.e., . Figure 3.c shows an example of the reconstructed mesh.
Ridge-aware reconstruction. Developable surfaces are ruled , i.e., contain straight lines on the surface as shown in Fig. 2. Our method exploits this geometric property as described in this section. Unlike existing parameterization-based methods which only handle smooth rulers, [26, 27, 2, 28], extracting arbitrary creases and ridges is more difficult when only sparse 3D points are available. We propose a sequential approach by first detecting ridges on the mesh that was obtained using our robust Poisson reconstruction method. After selecting the ridge candidates, we instantiate additional linear ridge constraints and incorporate them into the linear system that was solved earlier. This sequential approach is quite general and avoids overfitting. It also avoids spurious ridge candidates arising due to noise.
For each point on the mesh , we compute the Hessian as follows.
Based on the following Eigen decomposition of ,
we obtain principal curvatures and (
) and the corresponding eigenvectorsand .
The value of is equal to zero at all points on a developable surface. Thus, at any point , a straight line along direction must lie on the surface. As discussed earlier and shown in Fig. 2, the curvature along the ridge is zero while the curvature orthogonal to the ridge reaches a local extremum. We use this observation to select ridge candidates using the value of . Mesh points with greater than the threshold are selected as ridge candidates. (see Figure 3d for an example). The associated smoothness constraints in Eq. (4) are adjusted as follows.
where is the inner product and are orthonormal bases. is a convex monotonic function defined as , which places a greater weight along the ridge and smaller weight orthogonal to it. We also consider two more directional smoothness constraints similar to those stated in Eq. (12), for the two diagonal directions and .
3.2 Surface Unwrapping
Given the 3D surface reconstruction, our next step is to unwrap the surface. We take a conformal mapping approach to this problem, amongst which, Least Squares Conformal Mapping (LSCM) [11, 29] is a suitable choice. However, it is not resilient to the presence of outliers and susceptible to global distortion which can occur due to the absence of long-range constraints. We address both these issues and extend LSCM by incorporating an appropriate robustifier as well as ridge constraints to reduce global drift.
Conformal Mapping. For our mesh topology, each 3D point , on the grid on forms two triangles, one with its upper and left neighbor, the other with its lower and right neighbor on the grid. The triangulated 3D mesh is denoted as . A conformal map will produce an associated 2D mesh with the same connectivity but with 2D vertex positions such that the angle of all the triangles are best preserved. We denote the 2D mesh as , where .
For a particular 3D triangle with vertices at , , and , we seek its associated 2D vertex positions (, and ) under the conformal map. Using a local 2D coordinate basis for triangle , the conformality constraint is captured by the following linear equations.
Here, , is the area of , , and ( is similarly defined). Note that variables , , and were obtained from ’s vertex coordinates (see Fig. 4). Putting together the constraints for all the triangles, we have the following sparse linear system.
Using indices and to index the vertices and triangles respectively, the matrix in Equation (14) has the following non-zero entries.
Ridge constraints. Notice that the original conformal mapping has only local constraints, which will result in global distortion, Fig. 5.f. To reduce global distortions during unwrapping, we add ridge and boundary constraints to constrain the solution further.
We take into consideration two facts. First, the ridge lines remain straight after flattening but should essentially become invisible on the flattened surface. Second, the conformal mapping constraint Eq. (13) applies to beyond triangles. In particular, it is true for three collinear points. Therefore, we propose using the collinearity property to derive non-local constraints during flattening. and add it to our conformal map estimation problem. Referring to Fig. 4, and imagine the collinear case, that is when point is also lying on the axis; in such case, . In addition, the area of the triangle is zero. Hence, the ridge constraints can be written in a form similar to Eq. (13).
where are the targeted 2D coordinates similarly defined as . We select ridge candidates in the same way as we did earlier during reconstruction. However, this step is now more accurate because the surface is well reconstructed. For each ridge candidate (vertex), we find two farthest ridge candidates along the ridge line in opposite directions and instantiate the above mentioned constraint for the three points. We assume that the boundary of the flattened 2D document image has straight line segments (they need not be straight lines on the 3D surface). These boundary constraints can be expressed in a form similar to Eq. (16). We incorporate all ridge and boundary constraints into a system of linear equations.
where balances the ridge and boundary constraints. To avoid the trivial solution , we fix two points of to and . Equation (18) is then rewritten as
where is the energy function for the two fixed points. We solve the objective function using the iterative reweighted least squares method . Figure 5 shows a result from the conventional LSCM ( method) and our proposed method.
3.3 Implementation details
Sparse 3D reconstruction. We recover the initial sparse 3D point cloud using SfM. While any existing SfM method is applicable, we use the popular incremental SfM technique et al.  in our system. We typically capture five to ten still images for each document from different viewpoints. Capturing these images or equivalently a set of burst photos or a short video clip only takes a few seconds. Figure 1a shows an input example and the corresponding reconstruction.
Image warping. After recovering the flattened mesh grid , we unwrap the input image with the maximum document area in pixels. To obtain correspondence between the input image and , we project the 3D mesh points into the image to obtain image coordinates using the camera pose estimated using SfM. We then warp the image according to the correspondence between and
with bilinear interpolation.
We perform qualitative and quantitative evaluation on a wide variety of input documents. The first set of experiments show that our method can handle different paper types, document content and various types of folds and creases. Next, we report a quantitative evaluation based on known ground truth using local and global metrics where we demonstrate the superior performance and advantages of our method over existing methods [29, 30, 28]. In all the experiments, we set parameters as follows: , and . Our method is insensitive to these parameters. Varying by factors of 0.1 – 1.0 or varying or by 50% from these settings did not change the result significantly.
4.1 Test Data
The first six out of the 12 test sequences (I – VI) contain documents with no fold lines, one fold line, two to three parallel fold lines, and two to three crossing fold lines respectively. The other six sequences (VII – XII) contain documents with an increasing number of fold lines. Irregular fold lines were intentionally added to make the rectification more challenging. All documents were either placed on a planar or curved background surface. Sequence VII contains a shopping receipt on a paper roll whereas II and VIII contain pages from a book. Sequences III, IV, IX and X contain letters folded within envelopes. Sequence V, VI, XI and XII contain examples of documents folded inside a purse or notebook. The input images as well as the results from our method are shown in Fig. 6. Our method does not rely on the content, formatting, layout or color of the document. Thus, it is generally applicable as long as a sufficient number of sparse keypoints in the input images are available for SfM.
4.2 Quantitative Evaluation Metrics
We quantitatively evaluate the global and local distortion between the ground truth digital image and our rectified result using local and global metrics. The digital version of six out of the 12 test documents were available. We treat those images as ground truth and resize them by setting their height to 1000 pixels.
Global distortion metric. We first register the rectified image to the ground truth by estimating a global affine transform estimated using SIFT keypoint correspondences in these images .
This is achieved by minimizing the squared error.
where, and denote corresponding 2D keypoint positions using homogenous coordinates. We compute the global distortion metric as follows.
A perfect result has ; and larger values indicate more distortion (see comparative results in Fig. 7).
Local distortion metric. We compute the local metric by performing dense image registration using SIFT-flow  between the rectified image and the ground truth image. The frequency distribution of local displacements are shown in lower figure in Fig. 7 and compared with existing methods. We found dense registration to be more useful for an unbiased evaluation than sparse SIFT keypoint-based registration because sparse methods are more likely to ignore many matches if the result contains large deformations.
4.3 Comparison with existing methods
Perriollat et al. .
Their method explicitly parameterizes smooth rulers but cannot handle our document images with creases and folds.
Our method works fine on their dataset and produces a more accurate result than the one obtained by running their code
111 Their result shown here was generated by the original code provided by the authors. These result do not agree with the results in their paper. This is probably due to a difference in initialization.
Their result shown here was generated by the original code provided by the authors. These result do not agree with the results in their paper. This is probably due to a difference in initialization.(see Fig. 8). Although our result has minor artifacts due to self-occlusion and fore-shortening, the flattening result is quite accurate.
Brown et al. , Zhang et al. . We compare to both methods using our sequences where ground truth is available (Fig. 6). Since they require 3D range data, we use our reconstructed surface as their input and compare the surface flattening quality. We also compare our ridge-aware reconstruction to the standard Poisson reconstruction method. As shown in Fig. 7, the global and local distortion metrics introduced earlier are used in the evaluation. Our method has higher accuracy in terms of both metrics. Results from various methods have been compared in Fig. 5.
Evaluation on synthetic data. We compared our method with [29, 30] on synthetically generated dense 3D points because these methods require dense 3D points. We vary the point cloud size from 2K to 300K (common in 3D range data) and inject varying levels of Gaussian noise. The results from the three methods are compared in Figure 9. These experiments show that with low noise and high point density, all three methods are comparable in accuracy. However, when the points are sparser or when the noise level is higher, our method is more accurate than prior methods [29, 30].
5 Conclusion and Future Work
In this paper, we propose a method for automatically rectifying curved or folded paper sheets from a small number of images captured from different viewpoints. We use SfM to obtain sparse 3D points from images and propose ridge-aware surface reconstruction method which utilizes the geometric property of developable surface for accurate and dense 3D reconstruction of paper sheets. We also robustify the algorithms using optimization techniques. After recovering surface geometry, we unwrap the surface by adopting conformal mapping with both local and non-local constraints in a robust estimation framework. In the future we will address the correction of photometric inconsistencies in the document image caused by shading under scene illumination.
-  M. Pilu, “Undoing paper curl distortion using applicable surfaces,” in CVPR. IEEE, 2001.
-  G. Meng, Y. Wang, S. Qu, S. Xiang, and C. Pan, “Active flattening of curved document images via two structured beams,” in CVPR. IEEE, 2014.
-  M. S. Brown and W. B. Seales, “Image restoration of arbitrarily warped documents,” TPAMI, vol. 26, no. 10, pp. 1295–1306, 2004.
-  T. Wada, H. Ukida, and T. Matsuyama, “Shape from shading with interreflections under a proximal light source: Distortion-free copying of an unfolded book,” IJCV, vol. 24, no. 2, pp. 125–135, 1997.
-  L. Zhang, A. M. Yip, M. S. Brown, and C. L. Tan, “A unified framework for document restoration using inpainting and shape-from-shading,” Pattern Recognition, vol. 42, no. 11, pp. 2961–2978, 2009.
-  Y.-C. Tsoi and M. S. Brown, “Multi-view document rectification using boundary,” in CVPR. IEEE, 2007.
-  H. I. Koo, J. Kim, and N. I. Cho, “Composition of a dewarped and enhanced document image from two view images,” TIP, vol. 18, no. 7, pp. 1551–1562, 2009.
-  N. Stamatopoulos, B. Gatos, I. Pratikakis, and S. J. Perantonis, “Goal-oriented rectification of camera-based document images,” TIP, vol. 20, no. 4, pp. 910–920, 2011.
-  Z. Zhang, X. Liang, and Y. Ma, “Unwrapping low-rank textures on generalized cylindrical surfaces,” in ICCV. IEEE, 2011, pp. 1347–1354.
-  M. Kazhdan, M. Bolitho, and H. Hoppe, “Poisson surface reconstruction,” in Fourth Eurographics symposium on Geometry processing, 2006.
-  B. Lévy, S. Petitjean, N. Ray, and J. Maillot, “Least squares conformal maps for automatic texture atlas generation,” in ACM TOG, vol. 21. ACM, 2002, pp. 362–371.
-  Z. Zhang, C. Lim, and L. Fan, “Estimation of 3d shape of warped document surface for image restoration,” in ICPR, 2004.
-  C. L. Tan, L. Zhang, Z. Zhang, and T. Xia, “Restoring warped document images through 3d shape modeling,” TPAMI, vol. 28, no. 2, pp. 195–208, 2006.
-  F. Courteille, A. Crouzil, J.-D. Durou, and P. Gurdjos, “Shape from shading for the digitization of curved documents,” Machine Vision and Applications, vol. 18, no. 5, pp. 301–316, 2007.
-  Y.-C. Tsoi and M. S. Brown, “Geometric and shading correction for images of printed materials: a unified approach using boundary,” in CVPR. IEEE Computer Society, 2004.
-  A. Yamashita, A. Kawarago, T. Kaneko, and K. T. Miura, “Shape reconstruction and image restoration for non-flat surfaces of documents with a stereo vision system,” in ICPR. IEEE, 2004.
-  H. Cao, X. Ding, and C. Liu, “A cylindrical surface model to rectify the bound document image,” in ICCV, 2003.
-  H. Ezaki, S. Uchida, A. Asano, and H. Sakoe, “Dewarping of document image by global optimization,” in ICDAR. IEEE, 2005.
-  A. Ulges, C. H. Lampert, and T. M. Breuel, “Document image dewarping using robust estimation of curled text lines,” in ICDAR. IEEE, 2005.
-  S. Lu, B. M. Chen, and C. C. Ko, “A partition approach for the restoration of camera images of planar and curled document,” Image and Vision Computing, vol. 24, no. 8, pp. 837–848, 2006.
-  B. Fu, M. Wu, R. Li, W. Li, Z. Xu, and C. Yang, “A model-based book dewarping method using text line detection,” in 2nd Int. Workshop on Camera Based Document Analysis and Recognitionl, 2007.
-  G. Meng, C. Pan, S. Xiang, J. Duan, and N. Zheng, “Metric rectification of curved document images,” TPAMI, vol. 34, no. 4, pp. 707–722, 2012.
-  C. Liu, Y. Zhang, B. Wang, and X. Ding, “Restoring camera-captured distorted document images,” IJDAR, vol. 18, no. 2, pp. 111–124, 2014.
-  B. S. Kim, H. I. Koo, and N. I. Cho, “Document dewarping via text-line based optimization,” Pattern Recognition, 2015.
-  D. Salvi, K. Zheng, Y. Zhou, and S. Wang, “Distance transform based active contour approach for document image rectification,” in WACV. IEEE, 2015.
-  J. Liang, D. DeMenthon, and D. Doermann, “Geometric rectification of camera-captured document images,” TPAMI, vol. 30, no. 4, pp. 591–605, 2008.
-  Y. Tian and S. G. Narasimhan, “Rectification and 3d reconstruction of curved document images,” in CVPR. IEEE, 2011.
-  M. Perriollat and A. Bartoli, “A computational model of bounded developable surfaces with application to image-based three-dimensional reconstruction,” Computer Animation and Virtual Worlds, vol. 24, no. 5, pp. 459–476, 2013.
-  M. S. Brown, M. Sun, R. Yang, L. Yun, and W. B. Seales, “Restoring 2d content from distorted documents,” TPAMI, vol. 29, no. 11, pp. 1904–1916, 2007.
-  L. Zhang, Y. Zhang, and C. L. Tan, “An improved physically-based method for geometric restoration of distorted document images,” TPAMI, vol. 30, no. 4, pp. 728–734, 2008.
-  H. Avron, A. Sharf, C. Greif, and D. Cohen-Or, “ℓ 1-sparse reconstruction of sharp point set surfaces,” ACM TOG, 2010.
-  A. C. Öztireli, G. Guennebaud, and M. Gross, “Feature preserving point set surfaces based on non-linear kernel regression,” in Computer Graphics Forum, vol. 28, no. 2. Wiley Online Library, 2009, pp. 493–501.
-  R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of the Royal Statistical Society. Series B (Methodological), pp. 267–288, 1996.
-  E. J. Candes, M. B. Wakin, and S. P. Boyd, “Enhancing sparsity by reweighted minimization,” Journal of Fourier analysis and applications, vol. 14, no. 5-6, pp. 877–905, 2008.
-  E. Portnoy, “Developable surfaces in hyperbolic space,” Pacific Journal of Mathematics, vol. 57, no. 1, pp. 281–288, 1975.
-  R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press, ISBN: 0521540518, 2004.
-  D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” IJCV, vol. 60, no. 2, pp. 91–110, 2004.
-  C. Liu, J. Yuen, A. Torralba, J. Sivic, and W. T. Freeman, “Sift flow: Dense correspondence across different scenes,” in ECCV, 2008.