Image stitching has been well studied and widely applied in computer vision, where the most crucial step is to determine a 2D warp for each image and transform it into a common plane. Among these warps, similarity warp can preserve the angles between lines and maintain a uniform scale factor (ratio). Homography is the most flexible and popular global warp which preserves all straight lines, however, it introduces projective distortion at the same time. Basically, the projective distortion usually consists of distortion in vertical and horizontal. Many efforts have been devoted to mitigating these projective distortion. AutoStitch uses cylindrical warp as pre-processing step to address this problem, however, it inevitably curves straight lines and suffers from horizontal distortion. SPHP  takes advantage of similarity to maintain the original perspective of input image but suffers from line-bending. QH  uses the proposed quasi-homography warp to mitigate projective and perspective distortion simultaneously, but there still exists horizontal stretches and vertical distortion. In this letter, an effective algorithm based on the property of cylindrical warp is proposed to mitigate the projective distortion produced by homography. The strategy of image resizing for seam-cutting significantly decreases the processing time in different resolutions. Fig. 1 shows the flowchart of our proposed algorithm.
Ii Proposed half-cylindrical warp
Cylindrical warp  is often applied to image stitching to reduce the projective distortion produced by homography and get a multi-view panorama when the camera is assumed to be rotating around its vertical axis. A sketch of cylindrical warp is illustrated in Fig. 2. Generally, let be the focal length of the cylinder, then a plane 3D point is corresponding to the 3D cylindrical points , and the 2D point after cylindrical warp is computed by
Suppose to be the center of the cylinder, then the warping function can be rewritten as
Note that if is a fixed constant, then both and are linear functions of ,
Cylindrical warp maps the vertical line to (itself), which preserves the value of points and ratio of lengths on this line. In this manner, we can divide the image plane into two regions by the line , and separately manipulate homography warp and a composition of homography and cylindrical warps on each region.
Given an image with height and width , we let the upper-left point to be the origin, and -axis towards right, -axis towards down. The warping function of half-cylindrical warp is defined as
where denotes the homography warp, and are two regions divided by a line to apply different warps. The composition of homography and cylindrical warps efficiently mitigates the projective distortion in vertical, however, there is still some horizontal stretches or squeezes which makes the image unnatural, see Fig. 3.
Iii Pixel selection for ratio-preserving
The composition of homgraphy and cylindrical warps can only reduce the vertical distortion efficiently. A novel pixel selection strategy is proposed to preserve image width when employing warping function in the non-overlapping region. Without ambiguity, we use to denote original points, to denote points after homography, to denote points after composition of homography and cylindrical warps. Our strategy is proceeded by:
Using feature points to estimate a similarity warp
and compute the scale of pixel selection by , where and are the translation in horizontal and vertical respectively.
For any integer , selecting the points (where ) on horizontal lines among the image domain as sample points, where and is defined by
Recording these sampling points which are in the non-overlapping region after homography and cylindrical warps by .
Constraining the horizontal distance of these neighboring sample points to be one pixel in the final canvas, then applying the inversion map of cylindrical and homography warps on points and manipulating bilinear interpolation when the transformed coordinate is not an integer.
As we can see from the above procedures, the pixel selection strategy preserves the ratio as similarity in the non-overlapping region after . Moreover, the inversion of warps and interpolation make the results without blurring. Therefore, by combining warp and pixel selection strategy we finally get a more natural stitched result with less vertical and horizontal distortion.
Iv Parameter estimation
It is worth noting that there are some crucial parameters in Eq. (2,3), which should be carefully dealt with before applying half-cylindrical warp . In the following, we only consider horizontal stitching, vertical stitching can be obtained by a similar manner.
Firstly, we need to determine the abscissa of center. Since our final stitched result is divided by a line for applying different warps by its side, we set to be or (depending on the position of the target image). This line also divides the image into overlapping region and non-overlapping region, and the coordinate of homography and cylindrical warp on this line equals each other, thus we ensure the continuity of the stitched image.
Then, the ordinate of center should make the pair of corresponding points (e.g., and ) between vertical boundaries with minimal -offset, as shown in Fig. 4. We subsequently sample and track the points on the left and right boundaries of original image. Suppose and to be the ordinate of warped points under homography on the left and the right boundary respectively, then is estimated by
In fact, there exists a unique horizontal line that remains horizontal, i.e., the -offset equals , more details can be found in . Our estimation can be regarded as the discretization process of that method.
Finally, an appropriate focal length should be carefully estimated, since with too large or too small focal length can greatly affect the quality of the final stitched results. The primary purpose of our cylindrical warp is trying to make the image preserve the average height as original as possible after warping. For point after homography, the height in position after the composition of homgoraphy and cylindrical warps, denoted by , can be derived from Eq. (3)
where is the height after homography warp in (see the green line in Fig. 4). Then, we estimate focal length by solving the following non-linear error minimisation task
where is the height of desired average reference height after cylindrical warp. We set
to preserve the average height after homography. The selection of determines the focal length then further changes the shape of final results.
For naturalness, we compare our warp with homography, SPHP  and QH . Fig. 5 illustrates the naturalness comparison of stitched results, it is clear that our half-cylindrical ratio-preserving warp looks more natural than other warps in aspects of distortion in horizontal and vertical.
For efficiency, we resize the input images to of original size to speed up the time-consuming seam-cutting stage, then map the seam to original image for blending. Generally, our warp needs very little time, and the average total processing time for different resolutions of proposed method is significantly reduced as shown in Table I.
|resolution (pixel)||warp time (s)||total time (s)|
In this letter, a novel half-cylindrical ratio-preserving warp utilizing the property of cylindrical warp and a horizontal pixel selection strategy has been proposed to mitigate the projective distortion produced by homography. The parameter estimation stage further guarantees the robustness and naturalness of our warp. Experiments demonstrated that our warp can effectively mitigate projective distortion both in horizontal and vertical, and possesses high time efficiency.
-  Brown, M. and Lowe, D. G.: ‘Automatic panoramic image stitching using invariant features’, Int. J. Comput. Vis., 2007, 74, (1), pp. 59-73
-  Chang, C.-H., Sato, Y. and Chuang, Y.-Y.: ‘Shape-preserving half-projective warps for image stitching’, Proc. of IEEE CVPR, 2014, pp. 3254-3261
-  Li, N., Xu, Y. F. and Wang, C.: ‘Quasi-homography warps in image stitching’, IEEE Trans. Multimedia, 2017, online
-  Szeliski, R., and Shum, H.-Y.: ‘Creating full view panoramic image mosaics and environment maps’, Proc. of ACM SIGGRAPH, 1997, pp. 251-258
-  Szeliski, R.: ‘Computer Vision: Algorithms and Applications’, 2010
-  Lowe, D. G.: ’Distinctive image features from scale-invariant keypoints’, Int. J. Comput. Vis., 2004, 60, (2), pp. 91-110
-  Fischeler, M. A., and Bolles, R. C.: ’Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography’, Commun. ACM, 1981, 24, (6), pp. 381-395