In this article we devise a dynamic programming (DP) solution to dejittering problems with bounded displacements. In particular, we consider instances of the following image acquisition model. A -dimensional image defined on a two-dimensional domain is created by the equation
where is the original, undisturbed image, is a displacement perturbation, and denotes function composition. Typically, is a degradation generated by the acquisition process and is considered random. In addition, may exhibit additive noise . The central theme of this article is displacement error correction, which is to recover the original image solely from the corrupted image.
One particularly interesting class are dejittering problems. Jitter is a common artefact in digital images or image sequences and is typically attributed to inaccurate timing during signal sampling, synchronisation issues, or corrupted data transmission [10, 11]. Most commonly, it is observed as line jitter where entire lines are mistakenly shifted left or right by a random displacement. As a result, shapes appear jagged and unappealing to the viewer. Other—equally disturbing—defects are line pixel jitter and pixel jitter. The former type is due to a random shift of each position in horizontal direction only, while the latter is caused by a random displacement in . See Fig. 1 for examples.
The problem of dejittering is to reverse the observed effect and has been studied in numerous works, see [4, 6, 7, 10, 11, 13, 15, 16, 18]. Recent efforts either deal with finite-dimensional minimisation problems [11, 15, 16] or rely on an infinite-dimensional setting [4, 13, 18]. Typically, variational approaches such as [4, 13] are based on a linearisation of (1) and try to directly infer the true image. Alternatively, as done in , one can alternate between finding the displacement and inferring the original image. Both approaches typically enforce a certain regularity of the reconstructed image.
In this article, we investigate efficient solutions to dejittering models introduced in [4, 13]. However, we assume the magnitude of each component of the displacement , where , to be bounded by a constant . That is,
The main idea is to assume that (1 ) can be inverted (locally) by reconstructing the original value from a small neighbourhood.
Even though not guaranteed theoretically, this approach is found to work surprisingly well for Gaussian distributed displacements of zero mean.
A possible explanation is that the original value at a certain position
) can be inverted (locally) by reconstructing the original value from a small neighbourhood. Even though not guaranteed theoretically, this approach is found to work surprisingly well for Gaussian distributed displacements of zero mean. A possible explanation is that the original value at a certain positionis likely to occur in the close vicinity of . Moreover, it does not require derivatives of the disturbed data, which typically occur during linearisation of (1), see [4, 13]. The obvious drawback is that can only take values which appear in within a small neighbourhood of . As a result, its capabilities are limited in the presence of noise.
We build on previous work by Laborelli  and Nikolova [15, 16], and utilise DP for the numerical solution. For the removal of line jitter, we extend the algorithm in  to include regularisation of the displacement, yielding a stabler reconstruction. In comparison to the greedy approach in [15, 16] we are able to recover a global minimiser to the one-dimensional non-linear and possibly non-convex minimisation problem formulated in Sec. 2. For the case of line pixel jitter, we rewrite the problem into a series of independent minimisation problems, each of which can be solved optimally via DP. For pixel jitter removal we follow a different strategy as the regularisation term in the considered functional prohibits a straightforward decomposition into simpler subproblems. We employ block coordinate descent, which is an iterative method and is guaranteed to converge energy-wise . All of our algorithms generalise to -dimensional images defined on . Moreover, generalisation to regularisation functionals involving higher-order derivatives of the sought image and to higher-order discretisation accuracy is straightforward. Table 1 summarises the results of this work.
|Line jitter||same as  but with regularisation|
|Line pixel jitter|
|Pixel jitter||per-iteration complexity|
Let be a two-dimensional domain.
For , the -th power of the usual -norm of is denoted by .
For , we denote by , respectively, by the unknown original and the observed, possibly corrupted, -dimensional image.
A vector-valued function
-dimensional image. A vector-valued functionis given in terms of its components. We write for the -th partial derivative of with respect to and, for simplicity, we write for . For , the spatial gradient of in is and for it is given by the matrix . In the former case, its -norm is simply and in the latter case it is given by . For a function and we denote the -power of the norm of by . Moreover, denotes the essential supremum norm. A continuous image gives rise to a discrete representation of each pixel. A digital image is stored in matrix form , where denotes the number of columns arranged left to right and the number of rows stored from top to bottom.
2 Problem Formulation
Let be an observed and possibly corrupted image generated by (1). We aim to reconstruct an approximation of the original image . The main difficulty is that might not exist and that might exhibit noise. Lenzen and Scherzer  propose to find a minimising pair to the energy
such that satisfies (1). Here, is a regularisation functional and is a parameter. In what follows, we consider one exemplary class of displacements which arise in dejittering problems. They are of the form
with depending on the particular jitter model. Typically, is chosen in accordance with . We assume that is Gaussian distributed around zero with variance and whenever lies outside we typically have . In order to approximately reconstruct we will assume that, for every , there exists
In other words, we can invert (1) and locally reconstruct by finding . While this requirement trivially holds true for line jitter under appropriate treatment of the boundaries, it is not guaranteed in the cases of line pixel jitter and pixel jitter. Moreover, as a consequence of (4) and (2) we have, for ,
In this model, the corrupted image is assumed to be created as
where is a random displacement and is typically Gaussian white noise.
The corruption arises from a horizontal shift of each line by a random amount, resulting in visually unappealing, jagged shapes.
Assuming zero noise, (
is typically Gaussian white noise. The corruption arises from a horizontal shift of each line by a random amount, resulting in visually unappealing, jagged shapes. Assuming zero noise, (5) can be inverted within given . The original image is thus given by
For additional image denoising is required, see e.g. [17, Chap. 4] for standard methods of variational image denoising. We minimise the energy
subject to . The first term in (7) is suitable for displacements which are Gaussian distributed around zero. It prevents the reconstruction from being fooled by dominant vertical edges and effectively removes a constant additive displacement, resulting in a centred image. The second term utilises identity (6) and penalises the sum of the magnitudes of vertical derivatives of the reconstructed image up to -th order. Here, is a regularisation parameter and is an exponent. The proposed framework for the solution of (7) is more general and allows a different exponent and an individual weight for each term in the sum. Moreover, any other norm of might be considered.
We restrict ourselves to discretisations of and and assume that images are piecewise constant, are defined on a regular grid, and that all displacements are integer. Then, for every , we seek with . By discretising with backwards finite differences we obtain
Line Pixel Jitter.
Images degraded by line pixel jitter are generated by
where now depends on both and . As before, the displacement is in horizontal direction only. Images appear pixelated and exhibit horizontally fringed edges. In contrast to line jitter, in the noise-free setting one is in general not able to reconstruct the original image solely from , unless is bijective on for every . As a remedy, we utilise the fact that is assumed to be independent (in ) and identically Gaussian distributed around zero. The idea is that the original value , or a value sufficiently close, at is likely be found in a close neighbourhood with respect to the -direction. We assume that
exists and that . Clearly, it is not unique without further assumptions, however, finding one is sufficient. We utilise (11) and minimise
subject to . Again, and . In contrast to before, we decompose the objective into a series of minimisation problems, which then can be solved independently and in parallel by DP. To this end, let us rewrite
As before we consider derivatives up to second order. Assuming piecewise constant images defined on a regular grid we seek, for each , a displacement with . The finite-dimensional approximations of and hence read
An image corrupted by pixel jitter is generated by
where , , is now a vector-valued displacement. Edges in degraded images appear pixelated and fringed in both directions. Unless the displacement is bijective from to itself and , there is no hope that can be perfectly reconstructed from . However, we assume the existence of
such that , for . For and , we minimise
subject to , . In contrast to before, we only consider first-order derivatives of the sought image.
Assuming piecewise constant images on a regular grid and integer displacements, we seek for each an offset with . In further consequence, we obtain
3 Numerical Solution
Dynamic Programming on a Sequence.
Suppose we are given elements and we aim to assign to each element a label from its associated space of labels . Without loss of generality, we assume that all are identical and contain finitely many labels. A labelling is denoted by , where is the label assigned to the -th element.
Let us consider the finite-dimensional minimisation problem
We denote a minimiser by and its value by .
Here, is the penalty of assigning the label to element , whereas is the cost of assigning to the element and to , respectively.
Several minimisers of (17) might exist but finding one is sufficient for our purpose.
Energies (17 ) typically arise from the discretisation of computer vision problems such as one-dimensional signal denoising, stereo matching, or curve detection.
We refer to
) typically arise from the discretisation of computer vision problems such as one-dimensional signal denoising, stereo matching, or curve detection. We refer to for a comprehensive survey and to  for a general introduction to DP.
The basic idea for solving (17) is to restate the problem in terms of smaller subproblems. Let denote the cardinality of . Then, for and , we define as the minimum value of the above minimisation problem (17) over the first elements with the value of the last variable being set to the -th label. That is,
Moreover, we define for all and , and show the following recurrence:
Let denote the label of the -th element. Then,
By induction on the elements . Induction basis . Thus, holds. Inductive step. Assume it holds for . Then,
It is straightforward to see that Alg. 1 correctly computes all values of and hence the minimum value of (17). Its running time is in and its memory requirement in . Recovering a minimiser to (17) can be done either by a subsequent backward pass or even faster at the cost of additional memory by storing a minimising label in each iteration.
One can generalise the algorithm to energies involving higher-order terms of consecutive unknowns yielding a running time of and a memory requirement of . We refer to  for the details. The minimisation problems encountered in Sec. 2 involve terms of order at most three. It is straightforward to apply the above framework to (8), (9), (13), and (14).
Energy Minimisation on Graphs.
A more general point of view is to consider (17) on (undirected) graphs. Thereby, each element is associated with a vertex of the graph and one seeks a minimiser to
The first term sums over all vertices whereas the second term sums over all pairs of vertices which are connected via an edge in the graph. Such energies typically arise from the discretisation of variational problems on regular grids, such as for instance image denoising. In general, (18) is NP-hard. However, under certain restrictions on and , there exist polynomial-time algorithms. Whenever the underlying structure is a tree or a sequence as above, the problem can be solved by means of DP without further restrictions. See  for a general introduction to the topic.
Nevertheless, many interesting problems do fall in neither category. One remedy is block coordinate descent [2, 14] in order to approximately minimise (18), which we denote by . The essential idea is to choose in each iteration an index set which contains much less elements than , and to consider (18) only with respect to unknowns , . The values of all other unknowns are taken from the previous iteration. That is, one finds in each iteration
where denotes the energy in (18) with all unknowns not in taking the values from . Typically, is chosen such that (19) can be solved efficiently. It is straightforward to see that block coordinate descent generates a sequence of solutions such that never increases. The energy is thus guaranteed to converge to a local minimum with regard to the chosen .
We perform block coordinate descent for the solution of (16) and iteratively consider instances of (17).
Following the ideas in [2, 14] , we consecutively minimise in each iteration over all odd columns, all even columns, all odd rows, and finally over all even rows.
During minimisation the displacements of all other rows, respectively columns, are fixed.
, we consecutively minimise in each iteration over all odd columns, all even columns, all odd rows, and finally over all even rows. During minimisation the displacements of all other rows, respectively columns, are fixed. See Table1 for the resulting algorithms.
4 Numerical Results
We present qualitative results on the basis of a test image111Taken from BSDS500 dataset at https://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html and create image degraded by line jitter, line pixel jitter, and pixel jitter by sampling displacements from a Gaussian distribution with variance , and rounding them to the nearest integer, see Fig. 1. Vector-valued displacements are sampled component-wise. In addition, we create instances with additive Gaussian white noise of variance . We then apply to each instance the appropriate algorithm with varying parameters and , and show the results which maximise jitter removal, see Figs. 2, 3, and 4. The value of is set to the maximum displacement that occurred during creation. A Matlab/Java implementation is available online.222https://www.csc.univie.ac.at
While regularisation does not seem to be crucial for line jitter removal—apart from a centred image—for line pixel jitter and pixel jitter removal it is. Moreover, including second-order derivatives in our models does not necessarily improve the result. In all cases our results indicate effective removal of jitter, even in the presence of moderate noise. However, subsequent denoising seems obligatory.
5 Related work
Kokaram et al. [8, 9, 10] were among the first to consider line dejittering.
They employed a block-based autoregressive model for grey-valued images and developed an iterative multi-resolution scheme to estimate line displacements.
Subsequent drift compensation removes low frequency oscillations.
Their methods seek displacements which reduce the vertical gradients of the sought image.
were among the first to consider line dejittering. They employed a block-based autoregressive model for grey-valued images and developed an iterative multi-resolution scheme to estimate line displacements. Subsequent drift compensation removes low frequency oscillations. Their methods seek displacements which reduce the vertical gradients of the sought image.
For line jitter removal, a naive approach consists of fixing the first line of the image and successive minimisation of a mismatch function between consecutive lines. This greedy algorithm tends to introduce vertical lines in the reconstructed image and fails in the presence of dominant non-vertical edges [10, 11]. As a remedy, Laborelli  proposed to apply DP and to recover a horizontal displacement for each line by minimising the sum of the pixel-wise differences between two or three consecutive lines.
Shen  proposed a variational model for line dejittering in a Bayesian framework and investigated its properties for images in the space of bounded variation. In order to minimise the non-linear and non-convex objective, an iterative algorithm that alternatively estimates the original image and the displacements is devised.
Kang and Shen  proposed a two-step iterative method termed “bake and shake”.
In a first step, a Perona-Malik-type diffusion process is applied in order to suppress high-frequency irregularities in the image and to smooth distorted object boundaries.
In a second step, the displacement of each line is independently estimated by solving a non-linear least-squares problem.
In another article  , they investigated properties of slicing moments of images with bounded variation and proposed a variational model based on moment regularisation.
, they investigated properties of slicing moments of images with bounded variation and proposed a variational model based on moment regularisation.
In [15, 16], Nikolova considered greedy algorithms for finite-dimensional line dejittering with bounded displacements. These algorithms consider vertical differences between consecutive lines up to third-order and are applicable to grey-value as well as colour images. In each step, a non-smooth and possibly non-convex function is minimised by enumeration, leading to an algorithm. For experimental evaluation various error measures are discussed.
Lenzen and Scherzer [12, 13] considered partial differential equations for displacement error correction in multi-channel data.
Their framework is applicable to image interpolation, dejittering, and deinterlacing.
For line pixel dejittering they derived gradient flow equations for a non-convex variational formulation involving the total variation of the reconstructed image.
considered partial differential equations for displacement error correction in multi-channel data. Their framework is applicable to image interpolation, dejittering, and deinterlacing. For line pixel dejittering they derived gradient flow equations for a non-convex variational formulation involving the total variation of the reconstructed image.
In this article we presented efficient algorithms for line jitter, line pixel jitter, and pixel jitter removal in a finite-dimensional setting. By assuming (approximate) invertibility of the image acquisition equation we were able to cast the minimisation problems into a well-known DP framework. Our experimental results indicate effective removal of jitter, even in the presence of moderate noise.
-  A. Blake, P. Kohli, and C. Rother, editors. Markov Random Fields for Vision and Image Processing. MIT Press, 2011.
Q. Chen and V. Koltun.
Fast MRF optimization with application to depth reconstruction.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3914–3921. IEEE, June 2014.
-  T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to algorithms. MIT Press, Cambridge, MA, third edition, 2009.
-  G. Dong, A.R. Patrone, O. Scherzer, and O. Öktem. Infinite dimensional optimization models and PDEs for dejittering. In Scale Space and Variational Methods in Computer Vision. SSVM 2015. Lecture Notes in Computer Science, volume 9087, pages 678–689. Springer, April 2015.
-  P. F. Felzenszwalb and R. Zabih. Dynamic programming and graph algorithms in computer vision. IEEE Trans. Pattern Anal. Mach. Intell., 33(4):721–740, April 2011.
-  S. H. Kang and J. Shen. Video dejittering by bake and shake. Image Vision Comput., 24(2):143–152, 2006.
-  S. H. Kang and J. Shen. Image dejittering based on slicing moments. In X. C. Tai, K. A. Lie, T. F. Chan, and S. Osher, editors, Image Processing Based on Partial Differential Equations. Springer, 2007. Mathematics and Visualization.
-  A. Kokaram and P. Rayner. An algorithm for line registration of TV images based on a 2-D AR model. In Proceedings of Signal Processing VI, Theories and Applications, pages 1283–1286. European association for signal processing, 1992.
-  A. Kokaram, P. Rayner, P. Van Roosmalen, and J. Biemond. Line registration of jittered video. In Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 4, pages 2553–2556. IEEE Computer Society, 1997.
-  A. C. Kokaram. Motion Picture Restoration: Digital Algorithms for Artefact Suppression in Degraded Motion Picture Film and Video. Springer, London, UK, 1998.
-  L. Laborelli. Removal of video line jitter using a dynamic programming approach. In Proceedings of International Conference on Image Processing, 2003 (ICIP 2003), volume 2, pages II–331–334, Sept 2003.
-  F. Lenzen and O. Scherzer. A geometric PDE for interpolation of -channel data. In Scale Space and Variational Methods in Computer Vision. SSVM 2009. Lecture Notes in Computer Science, volume 5567, pages 413–425. Springer, 2009.
-  F. Lenzen and O. Scherzer. Partial differential equations for zooming, deinterlacing and dejittering. Int. J. Comput. Vision, 92(2):162–176, April 2011.
-  M. Menze, C. Heipke, and A. Geiger. Discrete optimization for optical flow. In Proceedings of the 37th German Conference on Pattern Recognition, volume 9358 of Lecture Notes in Computer Science, pages 16–28. Springer, 2015.
-  M. Nikolova. Fast dejittering for digital video frames. In Scale Space and Variational Methods in Computer Vision. SSVM 2009, volume 5567 of Lecture Notes in Computer Science, pages 439–451. Springer, 2009.
-  M. Nikolova. One-iteration dejittering of digital video images. J. Vis. Commun. Image Represent., 20:254–274, 2009.
-  O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier, and F. Lenzen. Variational methods in imaging. Number 167 in Applied Mathematical Sciences. Springer, 2009.
-  J. Shen. Bayesian video dejittering by BV image model. SIAM J. Appl. Math., 64(5):1691–1708, 2004.