Optical Flow on Evolving Sphere-Like Surfaces
In this work we consider optical flow on evolving Riemannian 2-manifolds which can be parametrised from the 2-sphere. Our main motivation is to estimate cell motion in time-lapse volumetric microscopy images depicting fluorescently labelled cells of a live zebrafish embryo. We exploit the fact that the recorded cells float on the surface of the embryo and allow for the extraction of an image sequence together with a sphere-like surface. We solve the resulting variational problem by means of a Galerkin method based on vector spherical harmonics and present numerical results computed from the aforementioned microscopy data.READ FULL TEXT VIEW PDF
Optical Flow on Evolving Sphere-Like Surfaces
Brightness and Mass Conservation Laws on Evolving Sphere-Like Surfaces
Motion estimation is a fundamental problem in image analysis and computer vision.
An important task within is optical flow computation.
It is concerned with the inference of a vector field describing the displacements of brightness patterns, such as moving objects, in a sequence of images.
Ever since the seminal work of Horn and Schunck
Motion estimation is a fundamental problem in image analysis and computer vision. An important task within is optical flow computation. It is concerned with the inference of a vector field describing the displacements of brightness patterns, such as moving objects, in a sequence of images. Ever since the seminal work of Horn and Schunck a variety of reliable and efficient methods have been proposed and successfully applied in a wide number of fields.
Primarily, optical flow is computed in the plane. However, it is readily generalised to non-Euclidean settings allowing, for instance, for cell motion analysis in time-lapse microscopy data. It has been only recently that high-resolution observations of biological model organisms such as the zebrafish became possible. Despite its importance for tissue and organ formation, little is known about cell migration and proliferation patterns during the zebrafish’s early embryonic development [1, 34]. Fluorescence microscopy nowadays allows to record time-lapse images on the scale of single cells, see e.g. [20, 28, 34]. Increasing spatial as well as temporal resolutions result in vast amounts of data, rendering extraction of information through visual inspection carried out by humans impracticable. Automated cell motion estimation therefore is key to large-scale analysis of such data. Optical flow computation delivers necessary quantitative methods and leads to insights into the underlying cellular mechanisms and the dynamic behaviour of cells. See, for example, [2, 29, 33, 34] and the references therein.
The primary biological motivation for this work is the desire to analyse cell motion in a living zebrafish during early embryogenesis. The data at hand depict endodermal cells expressing a green fluorescent protein. By virtue of laser-scanning microscopy, (volumetric time-lapse) 4D images of these labelled cells can be recorded without capturing the background. It is known that endodermal cells float on a so called monolayer during early embryonic development meaning that they do not stack on top of each other . Figure 1 depicts two frames of the captured sequence, containing only the upper hemisphere of the animal embryo. Observe the salient formation of the cells and the noise present in the images. More precisely, one can see the nuclei of cells forming a round surface in a single layer. For more details on the microscopy data we refer to Sec. 5.1.
We exploit this situation and model this layer as an evolving surface. A natural candidate for a parametrisation of such a zebrafish embryo is a sphere-like surface. It is topologically diffeomorphic to the 2-sphere and most commonly defined as the set of points
The function can be thought of as a radial deformation of and will have a dependence on time in the present paper. As a consequence, changes in the embryo’s geometry are attributed accordingly, albeit valid only during early stages of its development as cells tend to cluster subsequently. The main intention of this work is to conceive cell motion only on this moving 2-dimensional manifold. As a result we are able to reduce the spatial dimension of the data allowing for more efficient motion estimation in microscopy data. Figure 2 depicts two frames of the surface together with images obtained by restriction of the volumetric microscopy data in Fig. 1.
In this work we model the data as a time-dependent non-negative function . Its value directly corresponds to the fluorescence response of the observed cells. For a fixed time instant , the domain of is presumed to be a closed surface . We assume that this surface can be parametrised by a smooth radial map from the 2-sphere. The temporal evolution of the data can then be tracked by solving an optical flow problem on this moving surface or, more conveniently, an equivalent problem on the round sphere.
Traditionally, the starting point for optical flow is the assumption of constant brightness: a point moving along a trajectory does not change its intensity over time. On a moving domain one equivalently seeks, for every time , a tangent vector field that solves a generalised optical flow equation
at every point , where is the image sequence living on . Here, for a fixed time , denotes the (spatial) surface gradient, dot the standard inner product, and an appropriate temporal derivative.
The optical flow problem is ill-posed meaning that equation (1) is not uniquely solvable. A common approach to deal with non-uniqueness is Tikhonov regularisation, which consists of computing a minimiser of
The first term of the sum is usually the squared norm of the left-hand side of (1) and, in the present article, the second term will be an Sobolev norm.
The primary concern of this article is optical flow computation on evolving 2-dimensional Riemannian manifolds which can be parametrised from the sphere. Motivated by the aforementioned zebrafish microscopy data we consider closed surfaces for which the mapping
is a diffeomorphism between the 2-sphere and for every time . As a prototypical example we restrict ourselves to radially parametrised surfaces as they suit quite naturally to the given data.
The contributions of this work are as follows.
First, we give a variational formulation of optical flow on 2-dimensional closed Riemannian manifolds.
We assume a dependence on time and speak of evolving surfaces.
The main idea is to solve the problem by a Galerkin method in a finite-dimensional subspace of an appropriate (vectorial) Sobolev space.
We take advantage of the fact that tangential vector spherical harmonics form a complete orthonormal system for .
The sought vector field is thus uniquely determined when expanded in terms of the pushforward—by means of the differential of (2)—of these functions.
From that we arrive at a minimisation problem over , where is the dimension of the finite-dimensional space, and state the optimality conditions.
They can be written purely in terms of spherical quantities and solved on the 2-sphere.
To this end, we use a standard polyhedral approximation and locally interpolate spherical functions by piecewise quadratic polynomials.
For numerical integration we employ appropriate quadrature rules on the approximated sphere.
is the dimension of the finite-dimensional space, and state the optimality conditions. They can be written purely in terms of spherical quantities and solved on the 2-sphere. To this end, we use a standard polyhedral approximation and locally interpolate spherical functions by piecewise quadratic polynomials. For numerical integration we employ appropriate quadrature rules on the approximated sphere.
Second, to obtain the smooth sphere-like surface, which is described by the map (2), from the observed microscopy data, we formulate another variational problem on the sphere. The problem is essentially surface interpolation with Sobolev seminorm regularisation. Approximate cell centres serve as sample points of the surface. In particular, our microscopy data are supported only on the upper hemisphere, see Figs. 1 and 2. Scalar spherical harmonics are the appropriate choice for the numerical solution of the surface fitting problem, as they provide great flexibility with respect to the chosen space .
Finally, we present numerical experiments on the basis of the mentioned cell microscopy data of a live zebrafish. To this end we compute an approximation of the sphere-shaped embryo and obtain a sequence of images living on this moving surface. Eventually, we solve for the optical flow and present the results in a visually adequate manner.
The first variational formulation of optical flow is commonly attributed to Horn and Schunck . They attempted to compute a displacement field in by minimising a Tikhonov-regularised energy functional. It favours spatially regular vector fields by penalising its squared Sobolev seminorm. For introductory material on the subject we refer to [4, 5] and to  for a survey on various optical flow functionals. Well-posedness of the aforementioned energy was first shown by Schnörr . Moreover, there the problem was extended to irregular planar domains and solved by means of finite elements.
Weickert and Schnörr  considered a spatio-temporal model by extending the domain to . It additionally favours temporal regularity of the solution by including first derivatives with respect to time. Such models are of particular interest whenever trajectories are to be computed from the optical flow field. A unifying framework including several spatial as well as temporal regularisers was proposed in . For the purpose of evaluation and flow field visualisation a framework was created by Baker et al. .
Recently, generalisations to non-Euclidean domains have gained increasing attention. In  and  optical flow was considered in a spherical setting. Lefèvre and Baillet  adapted the Horn-Schunck functional to surfaces embedded in . Following Schnörr , they proved well-posedness of their formulation and employed a finite element method for solving the discrete problem on a triangle mesh. With an application to cell motion analysis, Kirisits et al. [22, 24] recently considered optical flow on evolving surfaces with boundary. They generalised the spatio-temporal model in  to a non-Euclidean and dynamic setting. Eventually, the problem was tackled numerically by solving the corresponding Euler-Lagrange equations in the coordinate domain. Similarly, Bauer et al.  studied optical flow on time-varying domains, with and without spatial boundary. They proposed a treatment on surfaces parametrised by product manifolds, constructed an appropriate Riemannian metric, and proved well-posedness of their formulation.
In Kirisits et al. , the authors considered various decomposition models for optical flow on the 2-sphere. The proposed functionals were solved by means of projection to a finite-dimensional space spanned by vector spherical harmonics. Concerning projection methods, Schuster and Weickert  solved the optical flow problem in solely based on regularisation by discretisation.
Regarding sphere-like surfaces and spherical harmonics expansion of closed surfaces we refer to  and the references therein.
Finally, let us mention [2, 29, 33, 34], where optical flow was employed for the analysis of cell motion in microscopy data. In particular, in Schmid et al.  the embryo of a zebrafish was modelled as a round sphere and motion of endodermal cells computed in map projections.
The remainder of this article is structured as follows. In Sec. 2, we formally introduce evolving sphere-like surfaces, recall the definition of vectorial Sobolev spaces on manifolds, and discuss both scalar and vector spherical harmonics on the 2-sphere. Section 3 is dedicated to optical flow on evolving surfaces and our variational formulation. In Sec. 4 we discuss the numerical solution. In particular, we propose to solve the resulting energy in a finite-dimensional subspace and rewrite the optimality conditions to be defined solely on the 2-sphere. Moreover, we show how to fit a sphere-like surface to the labelled cells in the microscopy data. Finally, in Sec. 5, we solve for the optical flow field and visualise the results. The appendix contains deferred material.
be the 2-sphere embedded in the 3-dimensional Euclidean space. The norm of , , is denoted by . By
we denote a smooth (local) parametrisation of mapping coordinates to points .
Furthermore, let denote a time interval and let be a family of closed smooth 2-manifolds . Each , , is assumed to be regular and oriented by the outward unit normal field , . We assume that (locally) admits a smooth parametrisation of the form
and call an evolving sphere-like surface.
We denote by a smooth function on the moving surface. Its coordinate representation and its corresponding spherical representation are given by
As a notational convention we indicate functions living on with a tilde and functions on with a hat, respectively. Their corresponding coordinate version is treated without special indication.
For convenience, we define smooth extensions of and to by
respectively. Note that, while is constant in the direction of the surface normal of , the extension in general is not. We point at Fig. 3 illustrating the setting.
Similarly, for vector-valued functions and the extensions to are defined component-wise and for all times . They are denoted by and , respectively. As a notational convention, boldface letters are used to denote vector fields. Moreover, we distinguish between lower and upper case boldface letters. The former identify tangent vector fields and their extensions to whereas the latter indicate general vector fields in .
For a differentiable function , we write as an abbreviation for the partial derivative of with respect to . That is, , where is the gradient of .
The tangent plane at a point is denoted by and the tangent bundle by . The orthogonal projector onto the tangent plane at , , is given by
In particular if , that is in (4) is identically one for all , the outward unit normal and the orthogonal projector are given by and , respectively.
In what follows, we define spatial differential operators. As they are identical to those on static surfaces we consider time arbitrary but fixed. Then, the surface gradient of , as given in (5), is defined by
where denotes the usual gradient of the embedding space. Let us stress that it is independent of the chosen extension, see e.g. [13, p. 389].
We emphasise that, in particular, if for all it follows that
The last term of the sum on the right hand side is the normal derivative of , which according to the definition of the extension in (6) vanishes. Thus,
For convenience let us observe that, by taking in (5), we arrive at
due to the chain rule and the projection onto the tangent plane
due to the chain rule and the projection onto the tangent plane.
Analogously to the surface gradient we define the spherical Laplace-Beltrami of as
where is the standard Laplacian of .
where is the parametrisation defined in (4), forms a basis of the tangent space at . Its elements form the gradient matrix , which is derived as follows.
By the chain rule,
Using (8) and the fact that equals on gives
where we have omitted the arguments and for better readability. Whenever convenient and no confusion will arise we will continue to do so.
By applying (9) backwards and the fact that we have shown
where is the gradient matrix associated with .
As a consequence, we can uniquely represent a tangent vector as , where is its coordinate representation, see e.g. [26, Prop. 3.15]. We call the components of .
In the sequel we will use Einstein summation convention. We sum over every index letter that appears exactly twice in an expression, once as a sub- and once as a superscript. For instance, we write for the sake of brevity.
where , , are functions obtained from the Gram-Schmidt process.
, are functions obtained from the Gram-Schmidt process.
Let us derive the following useful generalisation of (9). For a tangent vector , , the directional derivative of along at is
where the third equality follows from the first equation in (14). Analogously, for , with and , one can derive
Moreover, the coordinate representation of the surface gradient (8) is derived as follows. Let us start out with the first equation in (14). By writing in the coordinate basis, that is for some , we obtain from (14)
Multiplying with from the left yields
Furthermore, let be fixed and let . The surface integral of is
where is the Jacobian of . According to Theorem 3 in [11, p. 88], it is given by
and by using (12) yields
Note that and thus, terms of the form vanish. By the differentiability of , ones has
Therefore, , meaning that tangential and normal vectors are orthogonal. We emphasise that is commonly referred to as Riemannian metric. It is positive definite and thus, for all .
The differential of is a linear map and is given by
It follows from a direct calculation akin to the derivation of in (12).
Let us exhibit the action of , for and , onto a tangent vector . We have
In other words, the components are preserved whenever a tangent vector is mapped from to via the differential (20).
As a matter of fact, given a tangent vector field on , the differential gives rise to a unique tangent vector field on , see [26, Chapter 8]. Whenever we use and in the sequel we refer to their unique identification via the differential (20) and call the pushforward of . At this point, the reader might find it helpful to have a look at Fig. 5.
With the above definitions at hand we are able to relate the surface integral (18) to an integral on via a change of variables. The key is to compute a meaningful surface element as is the magnitude of the change of the volume element. The following lemma provides the required form.
Let be the standard parametrisation of ,
and let and be as above. Then, for ,
Let us denote by and the orthogonal unit vectors on in direction of and , respectively, which are obtained by normalising the coordinate basis . That is,
Moreover, a straightforward calculation gives
and thus, the surface gradient of in spherical coordinates (17) is given by
where we have replaced the coordinate basis with the orthonormal basis.
We briefly introduce the appropriate function spaces required for the variational optical flow formulation on Riemannian manifolds. Again, let us consider time arbitrary but fixed.
For a tangent vector field on we denote by the covariant derivative of at along the direction of a tangent vector . We define it as the tangential part of the usual directional derivative of the extension along in the embedding space, that is,
It is a linear operator and its Hilbert-Schmidt norm is given by
For each , we define the Sobolev space as the completion of the space of vector fields with respect to the norm
where the surface integral is defined in (18). Let us emphasise that (25) is indeed a norm whenever is diffeomorphic to the 2-sphere. The reason is that, by virtue of the Hairy Ball Theorem, there is no covariantly constant tangent vector field but , see e.g. [17, p. 125].
Alternatively, one can define Sobolev spaces of vector fields such that each component of a vector field originates from a scalar Sobolev space. See, for instance, Lefèvre and Baillet . On the 2-sphere, however, they are typically introduced by means of the spherical Laplace-Beltrami operator, see e.g. [30, Chapter 6.2] and Sec. 2.3 for the scalar counterpart. For a thorough treatment of Sobolev spaces on Riemannian manifolds we refer to the books [14, 38].
We denote by the space of homogeneous harmonic polynomials of degree with their domain restricted to . Its dimension is
We refer to Theorem 5.6 and Lemma 5.8 in [30, Sec. 5.1] for detailed proofs of the previous statements.
is a complete orthonormal system of with respect to the inner product on . In further consequence, for a function , we have the Fourier series representation
Again, we refer to [30, Sec. 5.1] for the proofs, in particular to Theorem 5.25. In the present article we employ fully normalised spherical harmonics. For the explicit construction see [30, Sec. 5.2].
Moreover, the norm of is readily stated in terms of the coefficients in the above expansion via Parseval’s identity
For an arbitrary real number , we define the Sobolev space as the completion of all functions with respect to the norm
We stress that, by (10), and we have for all yielding a sound definition. Accordingly, for , we define the seminorm of order by
Now that the space is endowed with a basis, we can proceed to define an orthonormal system for square integrable tangent vector fields on the sphere. This will immediately allow us to treat vector-valued problems consistently.
Let be a scalar spherical harmonic of degree . Any vector field that can be written in the form , where
is called a vector spherical harmonic of degree and type , cf. [12, Definition 5.2]. Recall that is the outward unit normal to .
By definition, is a normal field whereas and are tangent vector fields. Consequently, the latter are called tangential vector spherical harmonics. Note that, by means of the Hairy-Ball Theorem, no tangential vector spherical harmonics of degree zero exist.
In further consequence, let us denote by the space of square integrable tangent vector fields on equipped with the inner product
Here, denotes the usual spherical surface measure, see also Lemma 1.
Since (26) is an orthonormal set for , the set
is an orthonormal system for , where we have defined
for orthonormalisation purpose, see [12, Sec. 5.2]. Thus, every vector field can be written uniquely as
|family of sphere-like surfaces|
|tangent plane at|
|tangent plane at|
|outward unit normals to and|
|,||parametrisations of and|
|,||gradient matrix of and|
|orthonormal basis for|
|surface velocity of|
|smooth map from to and its differential|
|, ,||scalar function on , , and their coordinate version|
|,||surface gradient on and|
|, ,||tangent vector fields on , , and their coordinate version|
|covariant derivative of along direction on|
|scalar spherical harmonic of degree and order|
|vector spherical harmonic of degree , order , and type|