Optical Flow on Evolving Sphere-Like Surfaces

by   Lukas F. Lang, et al.

In this work we consider optical flow on evolving Riemannian 2-manifolds which can be parametrised from the 2-sphere. Our main motivation is to estimate cell motion in time-lapse volumetric microscopy images depicting fluorescently labelled cells of a live zebrafish embryo. We exploit the fact that the recorded cells float on the surface of the embryo and allow for the extraction of an image sequence together with a sphere-like surface. We solve the resulting variational problem by means of a Galerkin method based on vector spherical harmonics and present numerical results computed from the aforementioned microscopy data.



There are no comments yet.


page 28

page 29


A Numerical Framework for Efficient Motion Estimation on Evolving Sphere-Like Surfaces based on Brightness and Mass Conservation Laws

In this work we consider brightness and mass conservation laws for motio...

Decomposition of Optical Flow on the Sphere

We propose a number of variational regularisation methods for the estima...

Optical Flow on Evolving Surfaces with an Application to the Analysis of 4D Microscopy Data

We extend the concept of optical flow to a dynamic non-Euclidean setting...

Mapping the spatiotemporal dynamics of calcium signaling in cellular neural networks using optical flow

An optical flow gradient algorithm was applied to spontaneously forming ...

Two Methods for Iso-Surface Extraction from Volumetric Data and Their Comparison

There are various methods for extracting iso-surfaces from volumetric da...

Optimization on the Surface of the (Hyper)-Sphere

Thomson problem is a classical problem in physics to study how n number ...

Alignment of Microtubule Imagery

This work discusses preliminary work aimed at simulating and visualizing...

Code Repositories


Optical Flow on Evolving Sphere-Like Surfaces

view repo


Brightness and Mass Conservation Laws on Evolving Sphere-Like Surfaces

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Motion estimation is a fundamental problem in image analysis and computer vision. An important task within is optical flow computation. It is concerned with the inference of a vector field describing the displacements of brightness patterns, such as moving objects, in a sequence of images. Ever since the seminal work of Horn and Schunck 

[18] a variety of reliable and efficient methods have been proposed and successfully applied in a wide number of fields.

Primarily, optical flow is computed in the plane. However, it is readily generalised to non-Euclidean settings allowing, for instance, for cell motion analysis in time-lapse microscopy data. It has been only recently that high-resolution observations of biological model organisms such as the zebrafish became possible. Despite its importance for tissue and organ formation, little is known about cell migration and proliferation patterns during the zebrafish’s early embryonic development [1, 34]. Fluorescence microscopy nowadays allows to record time-lapse images on the scale of single cells, see e.g. [20, 28, 34]. Increasing spatial as well as temporal resolutions result in vast amounts of data, rendering extraction of information through visual inspection carried out by humans impracticable. Automated cell motion estimation therefore is key to large-scale analysis of such data. Optical flow computation delivers necessary quantitative methods and leads to insights into the underlying cellular mechanisms and the dynamic behaviour of cells. See, for example, [2, 29, 33, 34] and the references therein.

The primary biological motivation for this work is the desire to analyse cell motion in a living zebrafish during early embryogenesis. The data at hand depict endodermal cells expressing a green fluorescent protein. By virtue of laser-scanning microscopy, (volumetric time-lapse) 4D images of these labelled cells can be recorded without capturing the background. It is known that endodermal cells float on a so called monolayer during early embryonic development meaning that they do not stack on top of each other [39]. Figure 1 depicts two frames of the captured sequence, containing only the upper hemisphere of the animal embryo. Observe the salient formation of the cells and the noise present in the images. More precisely, one can see the nuclei of cells forming a round surface in a single layer. For more details on the microscopy data we refer to Sec. 5.1.

We exploit this situation and model this layer as an evolving surface. A natural candidate for a parametrisation of such a zebrafish embryo is a sphere-like surface. It is topologically diffeomorphic to the 2-sphere and most commonly defined as the set of points

The function can be thought of as a radial deformation of and will have a dependence on time in the present paper. As a consequence, changes in the embryo’s geometry are attributed accordingly, albeit valid only during early stages of its development as cells tend to cluster subsequently. The main intention of this work is to conceive cell motion only on this moving 2-dimensional manifold. As a result we are able to reduce the spatial dimension of the data allowing for more efficient motion estimation in microscopy data. Figure 2 depicts two frames of the surface together with images obtained by restriction of the volumetric microscopy data in Fig. 1.

Figure 1: Frames 70 (left) and 71 (right) of the volumetric zebrafish microscopy images recorded during early embryogenesis. The sequence contains a total number of 75 frames. Fluorescence response is indicated by blue colour and is proportional to the observed intensity. All dimensions are in micrometer (m).
Figure 2: Depicted are frames no. 70 (left) and 71 (right) of the processed zebrafish microscopy sequence. Top and bottom row differ by a rotation of 180 degrees around the -axis. All dimensions are in micrometer (m).

In this work we model the data as a time-dependent non-negative function . Its value directly corresponds to the fluorescence response of the observed cells. For a fixed time instant , the domain of is presumed to be a closed surface . We assume that this surface can be parametrised by a smooth radial map from the 2-sphere. The temporal evolution of the data can then be tracked by solving an optical flow problem on this moving surface or, more conveniently, an equivalent problem on the round sphere.

Traditionally, the starting point for optical flow is the assumption of constant brightness: a point moving along a trajectory does not change its intensity over time. On a moving domain one equivalently seeks, for every time , a tangent vector field that solves a generalised optical flow equation


at every point , where is the image sequence living on . Here, for a fixed time , denotes the (spatial) surface gradient, dot the standard inner product, and an appropriate temporal derivative.

The optical flow problem is ill-posed meaning that equation (1) is not uniquely solvable. A common approach to deal with non-uniqueness is Tikhonov regularisation, which consists of computing a minimiser of

The first term of the sum is usually the squared norm of the left-hand side of (1) and, in the present article, the second term will be an Sobolev norm.

1.1 Contributions

The primary concern of this article is optical flow computation on evolving 2-dimensional Riemannian manifolds which can be parametrised from the sphere. Motivated by the aforementioned zebrafish microscopy data we consider closed surfaces for which the mapping


is a diffeomorphism between the 2-sphere and for every time . As a prototypical example we restrict ourselves to radially parametrised surfaces as they suit quite naturally to the given data.

The contributions of this work are as follows. First, we give a variational formulation of optical flow on 2-dimensional closed Riemannian manifolds. We assume a dependence on time and speak of evolving surfaces. The main idea is to solve the problem by a Galerkin method in a finite-dimensional subspace of an appropriate (vectorial) Sobolev space. We take advantage of the fact that tangential vector spherical harmonics form a complete orthonormal system for . The sought vector field is thus uniquely determined when expanded in terms of the pushforward—by means of the differential of (2)—of these functions. From that we arrive at a minimisation problem over , where

is the dimension of the finite-dimensional space, and state the optimality conditions. They can be written purely in terms of spherical quantities and solved on the 2-sphere. To this end, we use a standard polyhedral approximation and locally interpolate spherical functions by piecewise quadratic polynomials. For numerical integration we employ appropriate quadrature rules on the approximated sphere.

Second, to obtain the smooth sphere-like surface, which is described by the map (2), from the observed microscopy data, we formulate another variational problem on the sphere. The problem is essentially surface interpolation with Sobolev seminorm regularisation. Approximate cell centres serve as sample points of the surface. In particular, our microscopy data are supported only on the upper hemisphere, see Figs. 1 and 2. Scalar spherical harmonics are the appropriate choice for the numerical solution of the surface fitting problem, as they provide great flexibility with respect to the chosen space .

Finally, we present numerical experiments on the basis of the mentioned cell microscopy data of a live zebrafish. To this end we compute an approximation of the sphere-shaped embryo and obtain a sequence of images living on this moving surface. Eventually, we solve for the optical flow and present the results in a visually adequate manner.

1.2 Related Work

The first variational formulation of optical flow is commonly attributed to Horn and Schunck [18]. They attempted to compute a displacement field in by minimising a Tikhonov-regularised energy functional. It favours spatially regular vector fields by penalising its squared Sobolev seminorm. For introductory material on the subject we refer to [4, 5] and to [40] for a survey on various optical flow functionals. Well-posedness of the aforementioned energy was first shown by Schnörr [35]. Moreover, there the problem was extended to irregular planar domains and solved by means of finite elements.

Weickert and Schnörr [42] considered a spatio-temporal model by extending the domain to . It additionally favours temporal regularity of the solution by including first derivatives with respect to time. Such models are of particular interest whenever trajectories are to be computed from the optical flow field. A unifying framework including several spatial as well as temporal regularisers was proposed in [41]. For the purpose of evaluation and flow field visualisation a framework was created by Baker et al. [6].

Recently, generalisations to non-Euclidean domains have gained increasing attention. In [19] and [37] optical flow was considered in a spherical setting. Lefèvre and Baillet [27] adapted the Horn-Schunck functional to surfaces embedded in . Following Schnörr [35], they proved well-posedness of their formulation and employed a finite element method for solving the discrete problem on a triangle mesh. With an application to cell motion analysis, Kirisits et al. [22, 24] recently considered optical flow on evolving surfaces with boundary. They generalised the spatio-temporal model in [42] to a non-Euclidean and dynamic setting. Eventually, the problem was tackled numerically by solving the corresponding Euler-Lagrange equations in the coordinate domain. Similarly, Bauer et al. [7] studied optical flow on time-varying domains, with and without spatial boundary. They proposed a treatment on surfaces parametrised by product manifolds, constructed an appropriate Riemannian metric, and proved well-posedness of their formulation.

In Kirisits et al. [23], the authors considered various decomposition models for optical flow on the 2-sphere. The proposed functionals were solved by means of projection to a finite-dimensional space spanned by vector spherical harmonics. Concerning projection methods, Schuster and Weickert [36] solved the optical flow problem in solely based on regularisation by discretisation.

Regarding sphere-like surfaces and spherical harmonics expansion of closed surfaces we refer to [32] and the references therein.

Finally, let us mention [2, 29, 33, 34], where optical flow was employed for the analysis of cell motion in microscopy data. In particular, in Schmid et al. [34] the embryo of a zebrafish was modelled as a round sphere and motion of endodermal cells computed in map projections.

The remainder of this article is structured as follows. In Sec. 2, we formally introduce evolving sphere-like surfaces, recall the definition of vectorial Sobolev spaces on manifolds, and discuss both scalar and vector spherical harmonics on the 2-sphere. Section 3 is dedicated to optical flow on evolving surfaces and our variational formulation. In Sec. 4 we discuss the numerical solution. In particular, we propose to solve the resulting energy in a finite-dimensional subspace and rewrite the optimality conditions to be defined solely on the 2-sphere. Moreover, we show how to fit a sphere-like surface to the labelled cells in the microscopy data. Finally, in Sec. 5, we solve for the optical flow field and visualise the results. The appendix contains deferred material.

2 Notation and Background

2.1 Sphere-Like Surfaces


be the 2-sphere embedded in the 3-dimensional Euclidean space. The norm of , , is denoted by . By


we denote a smooth (local) parametrisation of mapping coordinates to points .

Furthermore, let denote a time interval and let be a family of closed smooth 2-manifolds . Each , , is assumed to be regular and oriented by the outward unit normal field , . We assume that (locally) admits a smooth parametrisation of the form


and call an evolving sphere-like surface.

We denote by a smooth function on the moving surface. Its coordinate representation and its corresponding spherical representation are given by


As a notational convention we indicate functions living on with a tilde and functions on with a hat, respectively. Their corresponding coordinate version is treated without special indication.

For convenience, we define smooth extensions of and to by


respectively. Note that, while is constant in the direction of the surface normal of , the extension in general is not. We point at Fig. 3 illustrating the setting.

Figure 3: Schematic illustration of a cut through the surfaces and intersecting the origin. In addition, a radial line along which the extensions and are constant is shown. The surface normals are shown in grey.

Similarly, for vector-valued functions and the extensions to are defined component-wise and for all times . They are denoted by and , respectively. As a notational convention, boldface letters are used to denote vector fields. Moreover, we distinguish between lower and upper case boldface letters. The former identify tangent vector fields and their extensions to whereas the latter indicate general vector fields in .

For a differentiable function , we write as an abbreviation for the partial derivative of with respect to . That is, , where is the gradient of .

The tangent plane at a point is denoted by and the tangent bundle by . The orthogonal projector onto the tangent plane at , , is given by

In particular if , that is in (4) is identically one for all , the outward unit normal and the orthogonal projector are given by and , respectively.

In what follows, we define spatial differential operators. As they are identical to those on static surfaces we consider time arbitrary but fixed. Then, the surface gradient of , as given in (5), is defined by


where denotes the usual gradient of the embedding space. Let us stress that it is independent of the chosen extension, see e.g. [13, p. 389].

We emphasise that, in particular, if for all it follows that

The last term of the sum on the right hand side is the normal derivative of , which according to the definition of the extension in (6) vanishes. Thus,


For convenience let us observe that, by taking in (5), we arrive at


due to the chain rule and the projection onto the tangent plane


Analogously to the surface gradient we define the spherical Laplace-Beltrami of as


where is the standard Laplacian of .

The set


where is the parametrisation defined in (4), forms a basis of the tangent space at . Its elements form the gradient matrix , which is derived as follows.

Let be the extension of according to (6). Then, from (4) can be rewritten as

By the chain rule,

Using (8) and the fact that equals on gives

where we have omitted the arguments and for better readability. Whenever convenient and no confusion will arise we will continue to do so.

By applying (9) backwards and the fact that we have shown


where is the gradient matrix associated with .

As a consequence, we can uniquely represent a tangent vector as , where is its coordinate representation, see e.g. [26, Prop. 3.15]. We call the components of .

In the sequel we will use Einstein summation convention. We sum over every index letter that appears exactly twice in an expression, once as a sub- and once as a superscript. For instance, we write for the sake of brevity.

We underline that the coordinate basis (11) is not orthogonal in general. We will, however, require an orthonormal frame of the tangent space from Sec. 2.2 onwards. In the coordinate basis it reads


where ,

, are functions obtained from the Gram-Schmidt process.

Combining (5) and (9) with the expressions derived for and we can conveniently state that


Let us derive the following useful generalisation of (9). For a tangent vector , , the directional derivative of along at is


where the third equality follows from the first equation in (14). Analogously, for , with and , one can derive


As soon as we have established the relation between and it will conveniently allow us to switch between (15) and (16).

Moreover, the coordinate representation of the surface gradient (8) is derived as follows. Let us start out with the first equation in (14). By writing in the coordinate basis, that is for some , we obtain from (14)

Multiplying with from the left yields



Furthermore, let be fixed and let . The surface integral of is


where is the Jacobian of . According to Theorem 3 in [11, p. 88], it is given by

and by using (12) yields


Note that and thus, terms of the form vanish. By the differentiability of , ones has

Therefore, , meaning that tangential and normal vectors are orthogonal. We emphasise that is commonly referred to as Riemannian metric. It is positive definite and thus, for all .

The parametrisations and defined in (3) and (4), respectively, suggest the straightforward construction of a smooth map . It is given by the composition , that is

The differential of is a linear map and is given by


It follows from a direct calculation akin to the derivation of in (12).

Let us exhibit the action of , for and , onto a tangent vector . We have


In other words, the components are preserved whenever a tangent vector is mapped from to via the differential (20).

As a matter of fact, given a tangent vector field on , the differential gives rise to a unique tangent vector field on , see [26, Chapter 8]. Whenever we use and in the sequel we refer to their unique identification via the differential (20) and call the pushforward of . At this point, the reader might find it helpful to have a look at Fig. 5.

With the above definitions at hand we are able to relate the surface integral (18) to an integral on via a change of variables. The key is to compute a meaningful surface element as is the magnitude of the change of the volume element. The following lemma provides the required form.

Lemma 1.

Let be the standard parametrisation of ,

and let and be as above. Then, for ,


Let us denote by and the orthogonal unit vectors on in direction of and , respectively, which are obtained by normalising the coordinate basis . That is,


Moreover, a straightforward calculation gives

and thus, the surface gradient of in spherical coordinates (17) is given by

where we have replaced the coordinate basis with the orthonormal basis.

Using in (19), the Jacobian can be written as

Here, we have omitted the argument of . Then, the integral turns out to be

where the last equation follows from (18) if , the fact that , and

The concepts introduced above, and further properties thereof, may be found in any standard differential geometry book. For instance, in [9, 10, 25, 26].

2.2 Vectorial Sobolev Spaces on Manifolds

We briefly introduce the appropriate function spaces required for the variational optical flow formulation on Riemannian manifolds. Again, let us consider time arbitrary but fixed.

For a tangent vector field on we denote by the covariant derivative of at along the direction of a tangent vector . We define it as the tangential part of the usual directional derivative of the extension along in the embedding space, that is,


It is a linear operator and its Hilbert-Schmidt norm is given by


where denotes the orthonormal basis of the tangent space , cf. (13). We stress that (24) is invariant with respect to the chosen parametrisation.

For each , we define the Sobolev space as the completion of the space of vector fields with respect to the norm


where the surface integral is defined in (18). Let us emphasise that (25) is indeed a norm whenever is diffeomorphic to the 2-sphere. The reason is that, by virtue of the Hairy Ball Theorem, there is no covariantly constant tangent vector field but , see e.g. [17, p. 125].

Alternatively, one can define Sobolev spaces of vector fields such that each component of a vector field originates from a scalar Sobolev space. See, for instance, Lefèvre and Baillet [27]. On the 2-sphere, however, they are typically introduced by means of the spherical Laplace-Beltrami operator, see e.g. [30, Chapter 6.2] and Sec. 2.3 for the scalar counterpart. For a thorough treatment of Sobolev spaces on Riemannian manifolds we refer to the books [14, 38].

2.3 Spherical Harmonics

We denote by the space of homogeneous harmonic polynomials of degree with their domain restricted to . Its dimension is

An element , , is called a (scalar) spherical harmonic

. It is an infinitely often differentiable eigenfunction of the Laplace-Beltrami operator 

, defined in (10

), with corresponding eigenvalue

We refer to Theorem 5.6 and Lemma 5.8 in [30, Sec. 5.1] for detailed proofs of the previous statements.

The set


is a complete orthonormal system of with respect to the inner product on . In further consequence, for a function , we have the Fourier series representation

Again, we refer to [30, Sec. 5.1] for the proofs, in particular to Theorem 5.25. In the present article we employ fully normalised spherical harmonics. For the explicit construction see [30, Sec. 5.2].

Moreover, the norm of is readily stated in terms of the coefficients in the above expansion via Parseval’s identity

For an arbitrary real number , we define the Sobolev space as the completion of all functions with respect to the norm

We stress that, by (10), and we have for all yielding a sound definition. Accordingly, for , we define the seminorm of order by


Now that the space is endowed with a basis, we can proceed to define an orthonormal system for square integrable tangent vector fields on the sphere. This will immediately allow us to treat vector-valued problems consistently.

Let be a scalar spherical harmonic of degree . Any vector field that can be written in the form , where

is called a vector spherical harmonic of degree and type , cf. [12, Definition 5.2]. Recall that is the outward unit normal to .

By definition, is a normal field whereas and are tangent vector fields. Consequently, the latter are called tangential vector spherical harmonics. Note that, by means of the Hairy-Ball Theorem, no tangential vector spherical harmonics of degree zero exist.

In further consequence, let us denote by the space of square integrable tangent vector fields on equipped with the inner product

Here, denotes the usual spherical surface measure, see also Lemma 1.

Since (26) is an orthonormal set for , the set


is an orthonormal system for , where we have defined


for orthonormalisation purpose, see [12, Sec. 5.2]. Thus, every vector field can be written uniquely as

We refer to the books [12, 30] for further details on the subject. Table 1 contains a summary of notation used in the sequel.

coordinate domain
time interval
family of sphere-like surfaces
tangent plane at
tangent plane at
outward unit normals to and
, parametrisations of and
, gradient matrix of and
basis for
basis for
orthonormal basis for
surface velocity of
smooth map from to and its differential
, , scalar function on , , and their coordinate version
, surface gradient on and
, , tangent vector fields on , , and their coordinate version
covariant derivative of along direction on
scalar spherical harmonic of degree and order
vector spherical harmonic of degree , order , and type