Ratio-Preserving Half-Cylindrical Warps for Natural Image Stitching

03/18/2018
by   Yifang Xu, et al.
Nankai University
0

A novel warp for natural image stitching is proposed that utilizes the property of cylindrical warp and a horizontal pixel selection strategy. The proposed ratio-preserving half-cylindrical warp is a combination of homography and cylindrical warps which guarantees alignment by homography and possesses less projective distortion by cylindrical warp. Unlike previous approaches applying cylindrical warp before homography, we use partition lines to divide the image into different parts and apply homography in the overlapping region while a composition of homography and cylindrical warps in the non-overlapping region. The pixel selection strategy then samples the points in horizontal and reconstructs the image via interpolation to further reduce horizontal distortion by maintaining the ratio as similarity. With applying half-cylindrical warp and horizontal pixel selection, the projective distortion in vertical and horizontal is mitigated simultaneously. Experiments show that our warp is efficient and produces a more natural-looking stitched result than previous methods.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

page 3

01/27/2017

Quasi-homography warps in image stitching

Naturalness of warping is gaining extensive attention in image stitching...
03/17/2020

Hide Secret Information in Blocks: Minimum Distortion Embedding

In this paper, a new steganographic method is presented that provides mi...
03/25/2020

Fisheye Distortion Rectification from Deep Straight Lines

This paper presents a novel line-aware rectification network (LaRecNet) ...
02/13/2018

Single-Perspective Warps in Natural Image Stitching

Results of image stitching can be perceptually divided into single-persp...
03/09/2018

Image Registration Based Flicker Solving in Video Face Replacement and Analysis Based Sub-pixel Image Registration

In this paper, a framework of video face replacement is proposed and it ...
02/22/2021

Shadow Image Enlargement Distortion Removal

This project aims to adopt preprocessing operations to get less distorti...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Image stitching has been well studied and widely applied in computer vision, where the most crucial step is to determine a 2D warp for each image and transform it into a common plane. Among these warps, similarity warp can preserve the angles between lines and maintain a uniform scale factor (ratio). Homography is the most flexible and popular global warp which preserves all straight lines, however, it introduces projective distortion at the same time. Basically, the projective distortion usually consists of distortion in vertical and horizontal. Many efforts have been devoted to mitigating these projective distortion. AutoStitch

[1] uses cylindrical warp as pre-processing step to address this problem, however, it inevitably curves straight lines and suffers from horizontal distortion. SPHP [2] takes advantage of similarity to maintain the original perspective of input image but suffers from line-bending. QH [3] uses the proposed quasi-homography warp to mitigate projective and perspective distortion simultaneously, but there still exists horizontal stretches and vertical distortion. In this letter, an effective algorithm based on the property of cylindrical warp is proposed to mitigate the projective distortion produced by homography. The strategy of image resizing for seam-cutting significantly decreases the processing time in different resolutions. Fig. 1 shows the flowchart of our proposed algorithm.

Fig. 1: Overview of ratio-preserving half-cylindrical warp for image stitching

Ii Proposed half-cylindrical warp

Cylindrical warp [4] is often applied to image stitching to reduce the projective distortion produced by homography and get a multi-view panorama when the camera is assumed to be rotating around its vertical axis. A sketch of cylindrical warp is illustrated in Fig. 2. Generally, let be the focal length of the cylinder, then a plane 3D point is corresponding to the 3D cylindrical points , and the 2D point after cylindrical warp is computed by

Fig. 2: Illustration of cylindrical warp. (a) an image before and after projecting onto the cylinder by center . (b) a point from real world to in cylinder [5].
(1)

Suppose to be the center of the cylinder, then the warping function can be rewritten as

(2)
(3)

Note that if is a fixed constant, then both and are linear functions of ,

(4)

Cylindrical warp maps the vertical line to (itself), which preserves the value of points and ratio of lengths on this line. In this manner, we can divide the image plane into two regions by the line , and separately manipulate homography warp and a composition of homography and cylindrical warps on each region.

Given an image with height and width , we let the upper-left point to be the origin, and -axis towards right, -axis towards down. The warping function of half-cylindrical warp is defined as

(5)

where denotes the homography warp, and are two regions divided by a line to apply different warps. The composition of homography and cylindrical warps efficiently mitigates the projective distortion in vertical, however, there is still some horizontal stretches or squeezes which makes the image unnatural, see Fig. 3.

Fig. 3: before (left) and after (right) pixel selection in horizontal

Iii Pixel selection for ratio-preserving

The composition of homgraphy and cylindrical warps can only reduce the vertical distortion efficiently. A novel pixel selection strategy is proposed to preserve image width when employing warping function in the non-overlapping region. Without ambiguity, we use to denote original points, to denote points after homography, to denote points after composition of homography and cylindrical warps. Our strategy is proceeded by:

  1. Using feature points to estimate a similarity warp

    (6)

    and compute the scale of pixel selection by , where and are the translation in horizontal and vertical respectively.

  2. For any integer , selecting the points (where ) on horizontal lines among the image domain as sample points, where and is defined by

    (7)

    Recording these sampling points which are in the non-overlapping region after homography and cylindrical warps by .

  3. Constraining the horizontal distance of these neighboring sample points to be one pixel in the final canvas, then applying the inversion map of cylindrical and homography warps on points and manipulating bilinear interpolation when the transformed coordinate is not an integer.

As we can see from the above procedures, the pixel selection strategy preserves the ratio as similarity in the non-overlapping region after . Moreover, the inversion of warps and interpolation make the results without blurring. Therefore, by combining warp and pixel selection strategy we finally get a more natural stitched result with less vertical and horizontal distortion.

Iv Parameter estimation

It is worth noting that there are some crucial parameters in Eq. (2,3), which should be carefully dealt with before applying half-cylindrical warp . In the following, we only consider horizontal stitching, vertical stitching can be obtained by a similar manner.

Firstly, we need to determine the abscissa of center. Since our final stitched result is divided by a line for applying different warps by its side, we set to be or (depending on the position of the target image). This line also divides the image into overlapping region and non-overlapping region, and the coordinate of homography and cylindrical warp on this line equals each other, thus we ensure the continuity of the stitched image.

Fig. 4: Illustration of selecting with minimal -offset

Then, the ordinate of center should make the pair of corresponding points (e.g., and ) between vertical boundaries with minimal -offset, as shown in Fig. 4. We subsequently sample and track the points on the left and right boundaries of original image. Suppose and to be the ordinate of warped points under homography on the left and the right boundary respectively, then is estimated by

(8)

In fact, there exists a unique horizontal line that remains horizontal, i.e., the -offset equals , more details can be found in [3]. Our estimation can be regarded as the discretization process of that method.

Finally, an appropriate focal length should be carefully estimated, since with too large or too small focal length can greatly affect the quality of the final stitched results. The primary purpose of our cylindrical warp is trying to make the image preserve the average height as original as possible after warping. For point after homography, the height in position after the composition of homgoraphy and cylindrical warps, denoted by , can be derived from Eq. (3)

(9)

where is the height after homography warp in (see the green line in Fig. 4). Then, we estimate focal length by solving the following non-linear error minimisation task

(10)

where is the height of desired average reference height after cylindrical warp. We set

(11)

to preserve the average height after homography. The selection of determines the focal length then further changes the shape of final results.

V Experiments

In our experiments, we employ SIFT [6] to extract and match features, RANSAC [7] to estimate a global homography, and seam-cutting to blend the overlapping region.

For naturalness, we compare our warp with homography, SPHP [2] and QH [3]. Fig. 5 illustrates the naturalness comparison of stitched results, it is clear that our half-cylindrical ratio-preserving warp looks more natural than other warps in aspects of distortion in horizontal and vertical.

Fig. 5: Stitched result comparisons. We use blue rectangles to highlight curved lines and missing content, red ellipses to highlight horizontal distortion, and green lines to highlight vertical distortion

For efficiency, we resize the input images to of original size to speed up the time-consuming seam-cutting stage, then map the seam to original image for blending. Generally, our warp needs very little time, and the average total processing time for different resolutions of proposed method is significantly reduced as shown in Table I.

resolution (pixel) warp time (s) total time (s)
original after resizing
15002000 0.09 59.71 1.43
19202560 0.16 127.13 2.18
24483264 0.26 195.47 3.65
TABLE I: Comparisons of average processing time

Vi Conclusions

In this letter, a novel half-cylindrical ratio-preserving warp utilizing the property of cylindrical warp and a horizontal pixel selection strategy has been proposed to mitigate the projective distortion produced by homography. The parameter estimation stage further guarantees the robustness and naturalness of our warp. Experiments demonstrated that our warp can effectively mitigate projective distortion both in horizontal and vertical, and possesses high time efficiency.

References

  • [1] Brown, M. and Lowe, D. G.: ‘Automatic panoramic image stitching using invariant features’, Int. J. Comput. Vis., 2007, 74, (1), pp. 59-73
  • [2] Chang, C.-H., Sato, Y. and Chuang, Y.-Y.: ‘Shape-preserving half-projective warps for image stitching’, Proc. of IEEE CVPR, 2014, pp. 3254-3261
  • [3] Li, N., Xu, Y. F. and Wang, C.: ‘Quasi-homography warps in image stitching’, IEEE Trans. Multimedia, 2017, online
  • [4] Szeliski, R., and Shum, H.-Y.: ‘Creating full view panoramic image mosaics and environment maps’, Proc. of ACM SIGGRAPH, 1997, pp. 251-258
  • [5] Szeliski, R.: ‘Computer Vision: Algorithms and Applications’, 2010
  • [6] Lowe, D. G.: ’Distinctive image features from scale-invariant keypoints’, Int. J. Comput. Vis., 2004, 60, (2), pp. 91-110
  • [7] Fischeler, M. A., and Bolles, R. C.: ’Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography’, Commun. ACM, 1981, 24, (6), pp. 381-395