1 Introduction
In this article we devise a dynamic programming (DP) solution to dejittering problems with bounded displacements. In particular, we consider instances of the following image acquisition model. A dimensional image defined on a twodimensional domain is created by the equation
(1) 
where is the original, undisturbed image, is a displacement perturbation, and denotes function composition. Typically, is a degradation generated by the acquisition process and is considered random. In addition, may exhibit additive noise . The central theme of this article is displacement error correction, which is to recover the original image solely from the corrupted image.
One particularly interesting class are dejittering problems. Jitter is a common artefact in digital images or image sequences and is typically attributed to inaccurate timing during signal sampling, synchronisation issues, or corrupted data transmission [10, 11]. Most commonly, it is observed as line jitter where entire lines are mistakenly shifted left or right by a random displacement. As a result, shapes appear jagged and unappealing to the viewer. Other—equally disturbing—defects are line pixel jitter and pixel jitter. The former type is due to a random shift of each position in horizontal direction only, while the latter is caused by a random displacement in . See Fig. 1 for examples.
The problem of dejittering is to reverse the observed effect and has been studied in numerous works, see [4, 6, 7, 10, 11, 13, 15, 16, 18]. Recent efforts either deal with finitedimensional minimisation problems [11, 15, 16] or rely on an infinitedimensional setting [4, 13, 18]. Typically, variational approaches such as [4, 13] are based on a linearisation of (1) and try to directly infer the true image. Alternatively, as done in [18], one can alternate between finding the displacement and inferring the original image. Both approaches typically enforce a certain regularity of the reconstructed image.
In this article, we investigate efficient solutions to dejittering models introduced in [4, 13]. However, we assume the magnitude of each component of the displacement , where , to be bounded by a constant . That is,
(2) 
The main idea is to assume that (1
) can be inverted (locally) by reconstructing the original value from a small neighbourhood. Even though not guaranteed theoretically, this approach is found to work surprisingly well for Gaussian distributed displacements of zero mean. A possible explanation is that the original value at a certain position
is likely to occur in the close vicinity of . Moreover, it does not require derivatives of the disturbed data, which typically occur during linearisation of (1), see [4, 13]. The obvious drawback is that can only take values which appear in within a small neighbourhood of . As a result, its capabilities are limited in the presence of noise.We build on previous work by Laborelli [11] and Nikolova [15, 16], and utilise DP for the numerical solution. For the removal of line jitter, we extend the algorithm in [11] to include regularisation of the displacement, yielding a stabler reconstruction. In comparison to the greedy approach in [15, 16] we are able to recover a global minimiser to the onedimensional nonlinear and possibly nonconvex minimisation problem formulated in Sec. 2. For the case of line pixel jitter, we rewrite the problem into a series of independent minimisation problems, each of which can be solved optimally via DP. For pixel jitter removal we follow a different strategy as the regularisation term in the considered functional prohibits a straightforward decomposition into simpler subproblems. We employ block coordinate descent, which is an iterative method and is guaranteed to converge energywise [2]. All of our algorithms generalise to dimensional images defined on . Moreover, generalisation to regularisation functionals involving higherorder derivatives of the sought image and to higherorder discretisation accuracy is straightforward. Table 1 summarises the results of this work.
Algorithm  Time  Memory  Comment 

Line jitter  same as [11] but with regularisation  
Line pixel jitter  
Pixel jitter  periteration complexity 
Notation.
Let be a twodimensional domain. For , the th power of the usual norm of is denoted by . For , we denote by , respectively, by the unknown original and the observed, possibly corrupted,
dimensional image. A vectorvalued function
is given in terms of its components. We write for the th partial derivative of with respect to and, for simplicity, we write for . For , the spatial gradient of in is and for it is given by the matrix . In the former case, its norm is simply and in the latter case it is given by . For a function and we denote the power of the norm of by . Moreover, denotes the essential supremum norm. A continuous image gives rise to a discrete representation of each pixel. A digital image is stored in matrix form , where denotes the number of columns arranged left to right and the number of rows stored from top to bottom.2 Problem Formulation
Let be an observed and possibly corrupted image generated by (1). We aim to reconstruct an approximation of the original image . The main difficulty is that might not exist and that might exhibit noise. Lenzen and Scherzer [13] propose to find a minimising pair to the energy
(3) 
such that satisfies (1). Here, is a regularisation functional and is a parameter. In what follows, we consider one exemplary class of displacements which arise in dejittering problems. They are of the form
(4) 
with depending on the particular jitter model. Typically, is chosen in accordance with . We assume that is Gaussian distributed around zero with variance and whenever lies outside we typically have . In order to approximately reconstruct we will assume that, for every , there exists
In other words, we can invert (1) and locally reconstruct by finding . While this requirement trivially holds true for line jitter under appropriate treatment of the boundaries, it is not guaranteed in the cases of line pixel jitter and pixel jitter. Moreover, as a consequence of (4) and (2) we have, for ,
Line Jitter.
In this model, the corrupted image is assumed to be created as
(5) 
where is a random displacement and
is typically Gaussian white noise. The corruption arises from a horizontal shift of each line by a random amount, resulting in visually unappealing, jagged shapes. Assuming zero noise, (
5) can be inverted within given . The original image is thus given by(6) 
For additional image denoising is required, see e.g. [17, Chap. 4] for standard methods of variational image denoising. We minimise the energy
(7) 
subject to . The first term in (7) is suitable for displacements which are Gaussian distributed around zero. It prevents the reconstruction from being fooled by dominant vertical edges and effectively removes a constant additive displacement, resulting in a centred image. The second term utilises identity (6) and penalises the sum of the magnitudes of vertical derivatives of the reconstructed image up to th order. Here, is a regularisation parameter and is an exponent. The proposed framework for the solution of (7) is more general and allows a different exponent and an individual weight for each term in the sum. Moreover, any other norm of might be considered.
We restrict ourselves to discretisations of and and assume that images are piecewise constant, are defined on a regular grid, and that all displacements are integer. Then, for every , we seek with . By discretising with backwards finite differences we obtain
(8)  
(9) 
Line Pixel Jitter.
Images degraded by line pixel jitter are generated by
(10) 
where now depends on both and . As before, the displacement is in horizontal direction only. Images appear pixelated and exhibit horizontally fringed edges. In contrast to line jitter, in the noisefree setting one is in general not able to reconstruct the original image solely from , unless is bijective on for every . As a remedy, we utilise the fact that is assumed to be independent (in ) and identically Gaussian distributed around zero. The idea is that the original value , or a value sufficiently close, at is likely be found in a close neighbourhood with respect to the direction. We assume that
(11) 
exists and that . Clearly, it is not unique without further assumptions, however, finding one is sufficient. We utilise (11) and minimise
(12) 
subject to . Again, and . In contrast to before, we decompose the objective into a series of minimisation problems, which then can be solved independently and in parallel by DP. To this end, let us rewrite
As before we consider derivatives up to second order. Assuming piecewise constant images defined on a regular grid we seek, for each , a displacement with . The finitedimensional approximations of and hence read
(13)  
(14) 
Pixel Jitter.
An image corrupted by pixel jitter is generated by
(15) 
where , , is now a vectorvalued displacement. Edges in degraded images appear pixelated and fringed in both directions. Unless the displacement is bijective from to itself and , there is no hope that can be perfectly reconstructed from . However, we assume the existence of
such that , for . For and , we minimise
subject to , . In contrast to before, we only consider firstorder derivatives of the sought image.
Assuming piecewise constant images on a regular grid and integer displacements, we seek for each an offset with . In further consequence, we obtain
(16)  
3 Numerical Solution
Dynamic Programming on a Sequence.
Suppose we are given elements and we aim to assign to each element a label from its associated space of labels . Without loss of generality, we assume that all are identical and contain finitely many labels. A labelling is denoted by , where is the label assigned to the th element.
Let us consider the finitedimensional minimisation problem
(17) 
We denote a minimiser by and its value by . Here, is the penalty of assigning the label to element , whereas is the cost of assigning to the element and to , respectively. Several minimisers of (17) might exist but finding one is sufficient for our purpose. Energies (17
) typically arise from the discretisation of computer vision problems such as onedimensional signal denoising, stereo matching, or curve detection. We refer to
[5] for a comprehensive survey and to [3] for a general introduction to DP.The basic idea for solving (17) is to restate the problem in terms of smaller subproblems. Let denote the cardinality of . Then, for and , we define as the minimum value of the above minimisation problem (17) over the first elements with the value of the last variable being set to the th label. That is,
Moreover, we define for all and , and show the following recurrence:
Proposition 1.
Let denote the label of the th element. Then,
Proof.
By induction on the elements . Induction basis . Thus, holds. Inductive step. Assume it holds for . Then,
∎∎
It is straightforward to see that Alg. 1 correctly computes all values of and hence the minimum value of (17). Its running time is in and its memory requirement in . Recovering a minimiser to (17) can be done either by a subsequent backward pass or even faster at the cost of additional memory by storing a minimising label in each iteration.
One can generalise the algorithm to energies involving higherorder terms of consecutive unknowns yielding a running time of and a memory requirement of . We refer to [5] for the details. The minimisation problems encountered in Sec. 2 involve terms of order at most three. It is straightforward to apply the above framework to (8), (9), (13), and (14).
Energy Minimisation on Graphs.
A more general point of view is to consider (17) on (undirected) graphs. Thereby, each element is associated with a vertex of the graph and one seeks a minimiser to
(18) 
The first term sums over all vertices whereas the second term sums over all pairs of vertices which are connected via an edge in the graph. Such energies typically arise from the discretisation of variational problems on regular grids, such as for instance image denoising. In general, (18) is NPhard. However, under certain restrictions on and , there exist polynomialtime algorithms. Whenever the underlying structure is a tree or a sequence as above, the problem can be solved by means of DP without further restrictions. See [1] for a general introduction to the topic.
Nevertheless, many interesting problems do fall in neither category. One remedy is block coordinate descent [2, 14] in order to approximately minimise (18), which we denote by . The essential idea is to choose in each iteration an index set which contains much less elements than , and to consider (18) only with respect to unknowns , . The values of all other unknowns are taken from the previous iteration. That is, one finds in each iteration
(19) 
where denotes the energy in (18) with all unknowns not in taking the values from . Typically, is chosen such that (19) can be solved efficiently. It is straightforward to see that block coordinate descent generates a sequence of solutions such that never increases. The energy is thus guaranteed to converge to a local minimum with regard to the chosen .
We perform block coordinate descent for the solution of (16) and iteratively consider instances of (17). Following the ideas in [2, 14]
, we consecutively minimise in each iteration over all odd columns, all even columns, all odd rows, and finally over all even rows. During minimisation the displacements of all other rows, respectively columns, are fixed. See Table
1 for the resulting algorithms.4 Numerical Results
We present qualitative results on the basis of a test image^{1}^{1}1Taken from BSDS500 dataset at https://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html and create image degraded by line jitter, line pixel jitter, and pixel jitter by sampling displacements from a Gaussian distribution with variance , and rounding them to the nearest integer, see Fig. 1. Vectorvalued displacements are sampled componentwise. In addition, we create instances with additive Gaussian white noise of variance . We then apply to each instance the appropriate algorithm with varying parameters and , and show the results which maximise jitter removal, see Figs. 2, 3, and 4. The value of is set to the maximum displacement that occurred during creation. A Matlab/Java implementation is available online.^{2}^{2}2https://www.csc.univie.ac.at
While regularisation does not seem to be crucial for line jitter removal—apart from a centred image—for line pixel jitter and pixel jitter removal it is. Moreover, including secondorder derivatives in our models does not necessarily improve the result. In all cases our results indicate effective removal of jitter, even in the presence of moderate noise. However, subsequent denoising seems obligatory.
5 Related work
were among the first to consider line dejittering. They employed a blockbased autoregressive model for greyvalued images and developed an iterative multiresolution scheme to estimate line displacements. Subsequent drift compensation removes low frequency oscillations. Their methods seek displacements which reduce the vertical gradients of the sought image.
For line jitter removal, a naive approach consists of fixing the first line of the image and successive minimisation of a mismatch function between consecutive lines. This greedy algorithm tends to introduce vertical lines in the reconstructed image and fails in the presence of dominant nonvertical edges [10, 11]. As a remedy, Laborelli [11] proposed to apply DP and to recover a horizontal displacement for each line by minimising the sum of the pixelwise differences between two or three consecutive lines.
Shen [18] proposed a variational model for line dejittering in a Bayesian framework and investigated its properties for images in the space of bounded variation. In order to minimise the nonlinear and nonconvex objective, an iterative algorithm that alternatively estimates the original image and the displacements is devised.
Kang and Shen [6] proposed a twostep iterative method termed “bake and shake”. In a first step, a PeronaMaliktype diffusion process is applied in order to suppress highfrequency irregularities in the image and to smooth distorted object boundaries. In a second step, the displacement of each line is independently estimated by solving a nonlinear leastsquares problem. In another article [7]
, they investigated properties of slicing moments of images with bounded variation and proposed a variational model based on moment regularisation.
In [15, 16], Nikolova considered greedy algorithms for finitedimensional line dejittering with bounded displacements. These algorithms consider vertical differences between consecutive lines up to thirdorder and are applicable to greyvalue as well as colour images. In each step, a nonsmooth and possibly nonconvex function is minimised by enumeration, leading to an algorithm. For experimental evaluation various error measures are discussed.
considered partial differential equations for displacement error correction in multichannel data. Their framework is applicable to image interpolation, dejittering, and deinterlacing. For line pixel dejittering they derived gradient flow equations for a nonconvex variational formulation involving the total variation of the reconstructed image.
6 Conclusion
In this article we presented efficient algorithms for line jitter, line pixel jitter, and pixel jitter removal in a finitedimensional setting. By assuming (approximate) invertibility of the image acquisition equation we were able to cast the minimisation problems into a wellknown DP framework. Our experimental results indicate effective removal of jitter, even in the presence of moderate noise.
References
 [1] A. Blake, P. Kohli, and C. Rother, editors. Markov Random Fields for Vision and Image Processing. MIT Press, 2011.

[2]
Q. Chen and V. Koltun.
Fast MRF optimization with application to depth reconstruction.
In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, pages 3914–3921. IEEE, June 2014.  [3] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to algorithms. MIT Press, Cambridge, MA, third edition, 2009.
 [4] G. Dong, A.R. Patrone, O. Scherzer, and O. Öktem. Infinite dimensional optimization models and PDEs for dejittering. In Scale Space and Variational Methods in Computer Vision. SSVM 2015. Lecture Notes in Computer Science, volume 9087, pages 678–689. Springer, April 2015.
 [5] P. F. Felzenszwalb and R. Zabih. Dynamic programming and graph algorithms in computer vision. IEEE Trans. Pattern Anal. Mach. Intell., 33(4):721–740, April 2011.
 [6] S. H. Kang and J. Shen. Video dejittering by bake and shake. Image Vision Comput., 24(2):143–152, 2006.
 [7] S. H. Kang and J. Shen. Image dejittering based on slicing moments. In X. C. Tai, K. A. Lie, T. F. Chan, and S. Osher, editors, Image Processing Based on Partial Differential Equations. Springer, 2007. Mathematics and Visualization.
 [8] A. Kokaram and P. Rayner. An algorithm for line registration of TV images based on a 2D AR model. In Proceedings of Signal Processing VI, Theories and Applications, pages 1283–1286. European association for signal processing, 1992.
 [9] A. Kokaram, P. Rayner, P. Van Roosmalen, and J. Biemond. Line registration of jittered video. In Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 4, pages 2553–2556. IEEE Computer Society, 1997.
 [10] A. C. Kokaram. Motion Picture Restoration: Digital Algorithms for Artefact Suppression in Degraded Motion Picture Film and Video. Springer, London, UK, 1998.
 [11] L. Laborelli. Removal of video line jitter using a dynamic programming approach. In Proceedings of International Conference on Image Processing, 2003 (ICIP 2003), volume 2, pages II–331–334, Sept 2003.
 [12] F. Lenzen and O. Scherzer. A geometric PDE for interpolation of channel data. In Scale Space and Variational Methods in Computer Vision. SSVM 2009. Lecture Notes in Computer Science, volume 5567, pages 413–425. Springer, 2009.
 [13] F. Lenzen and O. Scherzer. Partial differential equations for zooming, deinterlacing and dejittering. Int. J. Comput. Vision, 92(2):162–176, April 2011.
 [14] M. Menze, C. Heipke, and A. Geiger. Discrete optimization for optical flow. In Proceedings of the 37th German Conference on Pattern Recognition, volume 9358 of Lecture Notes in Computer Science, pages 16–28. Springer, 2015.
 [15] M. Nikolova. Fast dejittering for digital video frames. In Scale Space and Variational Methods in Computer Vision. SSVM 2009, volume 5567 of Lecture Notes in Computer Science, pages 439–451. Springer, 2009.
 [16] M. Nikolova. Oneiteration dejittering of digital video images. J. Vis. Commun. Image Represent., 20:254–274, 2009.
 [17] O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier, and F. Lenzen. Variational methods in imaging. Number 167 in Applied Mathematical Sciences. Springer, 2009.
 [18] J. Shen. Bayesian video dejittering by BV image model. SIAM J. Appl. Math., 64(5):1691–1708, 2004.
Comments
There are no comments yet.