Validation of a PETSc based software implementing a 4DVAR Data Assimilation algorithm: a case study related with an Oceanic Model based on Shallow Water equation

10/02/2018 ∙ by Luisa Carracciuolo, et al. ∙ 0

In this work are presented and discussed some results related to the validation process of a software module based on PETSc which implements a Data Assimilation algorithm.



There are no comments yet.


page 3

page 6

page 22

page 23

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Definition of the DA problem and description of the PETSc based software implementation

The considered software intends to solve the problem defined on a suitable domain decomposition

of time-space domain as described in Definition 1 (all the needed notations can be founded in [2]).

Definition 1 (The 4D-VAR DA problem defined on the domain decomposition - the 4D-VAR DD-DA problem).

The 4D Variational DD-DA problem consists in computing the vector

such that




where the operator (the local 4D-VAR regularization functional) is defined as follows:


and where is a suitably defined operator on the overlapped domain . Parameter is a regularization parameter. The 4D-VAR regularization functional is defined as:


where is a regularization parameter, and () are the covariance matrices of the errors on the background and the observations respectively, while and denote the weighted euclidean norm.

The 4D-VAR DD-DA problem solution is computed performing the following steps on each subdomains (the so called 4D-VAR DD-DA algorithm):

  • Locally compute all the parameters that define the local 4D-VAR regularization functional

  • Locally compute the minimum (needed values for overlapping regions are obtained when necessary - i.e., for the model local evolution)

  • Globally contribute to computation of

In order to compute the minimum of all the functionals , the DD-DA algorithm has to face with some issues. In more details, we have to address:

  • the linearization of the operator , let us say , used for the evaluations of required by the minimisation algorithm;

  • the evaluation of the adjoint operator of , let us say , used for the evaluation of required by the minimisation algorithm;

Both the points above should require the computation of the discretization of the Jacobian of

Following some details about software implementation in PETSc (Portable, Extensible Toolkit for Scientific Computation)[7] environment. To implement the entire algorithm we plan to use:

  1. the PETSc time steppers TS module for solving time-dependent (nonlinear) PDEs, including the computation of adjoint;

  2. the PETSc DM module wich is a powerfull tool for the managment of all mesh data related with domain decomposition;

  3. The TAO software library [8] for the computation of (2). The Toolkit for Advanced Optimization (TAO) is aimed at the solution of large-scale optimization problems on high-performance architectures. TAO is suitable for both single-processor and massively-parallel architectures. The current version of TAO has algorithms for unconstrained and bound-constrained optimization.

  4. The SLEPc software library [9] for the computation of spectral decomposition usefull to compute a preconditioner of the error covariance matrices (i.e., see approach used in [2]

    ). The Scalable Library for Eigenvalue Problem Computations (SLEPc) is a software library for the solution of large scale sparse eigenvalue problems on parallel computers. It can also be used for computing a partial SVD of a large, sparse, rectangular matrix, and to solve nonlinear eigenvalue problems.

All the abobe mentioned software are integrated or based on PETSc (see figure 1 for a representation of the Software stack and algorithm implementation).

To represent the Jacobian of in PETSc we decided to follow a “matrix-free approach”: we used a MATHSHELL type for PETSc Mat object to represent just defining its way of operating

Figure 1: Representation of the Software stack and algorithm implementation

2 Case study description

The case study is based on the Shallow Water Equations (SWEs) on the sphere. The SWE have been used extensively as a simple model of the atmosphere or ocean circulation since they contain the essential wave propagation mechanisms found in general circulation models [1]. The SWEs in spherical coordinates are:


Here is the Coriolis parameter given by , where is the angular speed of the rotation of the Earth, is the height of the homogeneous atmosphere (or of the free ocean surface), and are the zonal and meridional wind (or the ocean velocity) components, respectively, and are the latitudinal and longitudinal directions, respectively, is the radius of the earth and is the gravitational constant.

We express the system of equations (5)-(7) using a compact form, i.e.:






We discretize (8) just in space using an un-staggered Turkel-Zwas scheme [5, 6], and we obtain:






If we define as follows:


we note that the symbol represents the model “applied” times.

We also note that the numerical model is defined by the following parameters:

  • discretization step in time domain,

  • , discretization step in space domain,

  • parameter of the Turkel-Zwas schema,

  • , parameters of the Turkel-Zwas schema.

To verify the correct operation of the software module which implements the model, we tested the computed values of , when (i.e., when domain is not decomposed), where:

  1. ,

  2. ,

  3. and ,

  4. is a syntetic vector containing all the considered fields: the see-level field is generated by a Gaussian stochastic process; both velocity fields and are set to zero,

  5. e defined on the basis of discretization grid used by data available at repository Ocean Synthesis/Reanalysis Directory of Hamburg University (see [3]).

We note that the values for parameters at above mentioned points 1, 2 and 3 were chosen on the basis of the considerations and results described in [4].

In figure 2-(a) is represented. In figure 2-(b)-(d) are represented respectively where . We used different values for with the aim to empirically determine the “best value” for . The considered values for were chosen taking into account the considerations about CFL condition for Turkel-Zwas methods described in [4].

To give a measure of how differs from depending from , in table 1 we show

for the considered values of .

50.0 2.493588e-03
100.0 4.846299e-03
150.0 6.949851e-03
200.0 8.737334e-03
Table 1: The values of as function of .
(a) - Initial guess for the model

(b) - (c) -
Figure 2: ()

3 Test results related with the DA software module operation

To verify the correct operation of the software module which implements the DA process we tested the computed values of , when (i.e., when domain is not decomposed), starting:

  • from the () diagonal matrix representing the covariance matrix of the errors on all the observations vectors

  • from the diagonal matrix representing the observational operator and

  • from the backgroud and

  • from the set of observations vectors ,

  • and by using, as preconditioner of covariance matrix , its Truncated SVD (the first singular values of are considered),


  1. ,

  2. Problem 1 For each ,

    (each elements of were obtained by rounding, on the third significant digit, the respective elements of ).

    is the identity matrix.

    Problem 2 For each , if is a multiple of
    ( are sparse vectors whose lenght is and whose non-zero elements (the of the total) were obtained by adding a scaled number

    from normal distribution to the respective elements of

    Problem 3 For each ,
    (each elements of were obtained by rounding, on the second significant digit, the respective elements of ). is the identity matrix.
    Problem 4 For each , if is a multiple of
    ( are sparse vectors whose lenght is and whose non-zero elements (the of the total) were obtained by adding, to the respective elements of , a number from normal distribution scaled by a factor related with order of magnitude of each elements ).

  3. ,

  4. .

  5. .

  6. , where

    and where

Figure 3 shows how the background (the red circle in the image) and observations (the blue circle in the image) were chosen/built from data: in particular, for each values of , the image intends to show which subset of is considered to generate background and observations. In particular, the procedure used to build the input data for DA problem performs the following steps:

  1. “application” of the model to the starting point to obtain the vector where ;

  2. from the vector computed at above point 1 we obtain both the background vector and, by using one of the definitions for Problem 1 or Problem 2, the first observation vector ;

  3. further “application” of the model to compute the set of vectors from which obtain, by using one of the definitions for Problem 1 or Problem 2, the remaining observation vectors .

Then, the “assimilation window” in time domain, when the value of is fixed, is the intervall defined as:

Figure 3: How and were chosen/built from data

Three sets of tests are performed: the first set intends to evaluate how different values for and influence the behavior of DA software module; the second and third sets of tests intend to give elements to evaluate how Data Assimilation used during model application (i.e., ) improve the quality of the model computed values without DA (i.e., ) with respect to observations 111We note that the symbols and represent the model “applied” times to the background and to the solution of DA problem respectively.

Tests Set 1 In order to evaluate how different values for and influence the behavior of DA software module, in tables 3 (for Problem 1), 4 (for Problem 2), 5 (for Problem 3) and 6 (for Problem 4) the values of and are showed for the above listed values of and and for the four considered problems where


and where . We note that values of are not reported when the algorithm failed (i.e., when the Truncated SVD computation failed). The table 2 show, as examples, the values of when and .

50.0 100.0 150.0 200.0
00 1.489436e+05 1.489436e+05 1.489436e+05 1.489436e+05
01 5.922436e-12 5.918084e-12 1.421180e-11 1.518229e-07
02 1.098639e-18 3.745277e-12 1.058226e-11 3.045052e-11
03 2.717045e-22 2.956363e-12 5.891872e-12 2.154711e-11
04 7.077415e-23 2.450420e-12 5.477584e-12 1.815192e-11
05 1.984375e-23 2.034103e-12 4.955457e-12 1.770864e-11
06 1.308422e-25 1.248364e-12 4.426257e-12 1.447040e-11
07 6.804022e-26 9.572109e-13 4.043577e-12 1.247191e-11
08 6.736329e-26 9.029654e-13 3.518900e-12 1.176448e-11
09 6.642977e-26 7.345570e-13 2.587594e-12 1.014551e-11
10 6.614099e-26 7.164269e-13 2.226629e-12 9.308843e-12
11 5.977310e-28 5.974983e-13 1.710227e-12 5.925403e-12
Table 2: The first singular values of matrix when ().

We also note that:

  1. the smaller values for is (i.e., where ), more the DA software module is able to effectively compute the solution of the DA problem;

  2. the larger values for is, more accurate is the solution of the DA problem computed by DA software module (if it is successful).

This behavior could be explained considering that:

  1. the smaller the value of is, better conditioned is the matrix , but also

  2. the larger values for is, “closer” the matrices and are.

5.923000468033734e-03 5.918233049175874e-03 5.909744367520904e-03 5.901306860071136e-03 5.893094645517731e-03 5.884752741767643e-03 5.923000468033728e-03 5.918233049175878e-03 5.909744367520882e-03 5.901306860071130e-03 5.893094645517735e-03 5.884752741767650e-03 5.923000468033734e-03 5.918233049175892e-03 5.893094645517728e-03 5.884752741767642e-03 5.923000468033734e-03 5.918233049175873e-03 5.909744367520906e-03 5.901306860071137e-03 5.893094645517726e-03 5.884752724034771e-03 5.923000468033739e-03 5.918233049175869e-03 5.909744367520902e-03 5.901306860071123e-03 5.893094612088676e-03 5.884752741767648e-03 5.918233049175874e-03 5.909744377775199e-03 5.901306706186256e-03
6.084454299095919e-03 6.073321261976583e-03 6.052230870989613e-03 6.028625075904369e-03 6.006605050609536e-03 5.984624179759802e-03   6.084454299095923e-03 6.073321261976588e-03 6.052230870989617e-03 6.028625075904372e-03 6.006605050609541e-03 5.984624179759798e-03 6.084454299095920e-03 6.073321261976588e-03 6.052230870989613e-03 6.006605050609541e-03 5.984624179759808e-03 6.084454298277842e-03 6.073321261976589e-03 6.052230870989612e-03 6.028625162320382e-03 6.006594251561000e-03 5.984624177098236e-03 6.084454340791144e-03 6.052230870989642e-03 6.028625075904334e-03 6.084454255839374e-03 6.073321261494072e-03 6.052230870989607e-03 6.028625075885917e-03
6.280754455601023e-03 6.263201134463331e-03 6.225669222102765e-03 6.184831399882742e-03 6.143615055490961e-03 6.102356895266559e-03 6.280754455601023e-03 6.263201134463330e-03 6.225669222102762e-03 6.184831399882739e-03 6.143615055490959e-03 6.102356895266554e-03 6.280754455601023e-03 6.263201134463330e-03 6.225669222102764e-03 6.184831399882742e-03 6.102356895266563e-03 6.280754455601023e-03 6.263201064275920e-03 6.184831399882738e-03 6.143615055490968e-03 6.102356895266562e-03 6.280754446106453e-03 6.263201055314105e-03 6.225669243979252e-03 6.143615055490962e-03 6.102356790788719e-03 6.280754387072876e-03 6.263201117930033e-03 6.225668830117284e-03 6.143615138972201e-03 6.102356811770067e-03
6.462142195081026e-03 6.437375271097644e-03 6.392026891087157e-03 6.340783651666366e-03 6.292150917739605e-03 6.244798377227290e-03 6.462142195081031e-03 6.437375271097648e-03 6.392026891087095e-03 6.340783651666372e-03 6.292150917739616e-03 6.244798377227289e-03 6.462142195081022e-03 6.392026891087161e-03 6.340783652553978e-03 6.292150917739606e-03 6.244798377227294e-03 6.437375271097721e-03 6.392026891087158e-03 6.340783669355231e-03 6.292150917739599e-03 6.244798377227293e-03 6.437375294652374e-03 6.292150917739615e-03 6.437375306111083e-03 6.392026896528792e-03 6.340783515516571e-03
Table 3: and as function of and () - Problem 1
3.453467033333584e-02 3.496046426389585e-02 3.461814248792758e-02 3.496047407097717e-02 3.461815048663301e-02 3.496048249133105e-02 3.453467033333583e-02 3.496046426389589e-02 3.461814248792760e-02 3.496047407097717e-02 3.461815048663298e-02 3.496048249133105e-02 3.453467033333584e-02 3.496046426389582e-02 3.461815048663301e-02 3.496048249133105e-02 3.453467033333584e-02 3.496046426389584e-02 3.461814248792756e-02 3.496047407097716e-02 3.461815048663301e-02 3.496048237755320e-02 3.453467033333585e-02 3.496046426389584e-02 3.461814248792758e-02 3.496047407097717e-02 3.461815029375775e-02 3.496048249133106e-02 3.496046426389585e-02 3.461814250679108e-02 3.496047385347141e-02
3.453457238151639e-02 3.496035264638818e-02 3.461805630410621e-02 3.496039039880471e-02 3.461808760666546e-02 3.496042291664354e-02 3.453457238151639e-02 3.496035264638819e-02 3.461805630410621e-02 3.496039039880478e-02 3.461808760666545e-02 3.496042291664356e-02 3.453457238151640e-02 3.496035264638819e-02 3.461805630410621e-02 3.461808760666545e-02 3.496042291664352e-02 3.453457249154981e-02 3.496035264638817e-02 3.461805630410621e-02 3.496039035077939e-02 3.461809290479863e-02 3.496042290325619e-02 3.453457274084655e-02 3.461805630410599e-02 3.496039039880472e-02 3.453457237819704e-02 3.496035261803864e-02 3.461805630410619e-02 3.496039039846557e-02
3.453440979970871e-02 3.496017506248329e-02 3.488429733019234e-02 3.483606155320147e-02 3.461798505691022e-02 3.496032646744604e-02 3.453440979970870e-02 3.496017506248329e-02 3.488429733019234e-02 3.483606155320147e-02 3.461798505691021e-02 3.496032646744603e-02 3.453440979970870e-02 3.496017506248329e-02 3.488429733019234e-02 3.483606155320146e-02 3.496032646744604e-02 3.453440979970870e-02 3.496017499975523e-02 3.483606155320146e-02 3.461798505691016e-02 3.496032646744601e-02 3.453440976085218e-02 3.496017512611298e-02 3.488429724470573e-02 3.461798505691022e-02 3.496032641471299e-02 3.453440988054077e-02 3.496017506439134e-02 3.488429726668655e-02 3.461798464694495e-02 3.496032654977042e-02
3.453419647204346e-02 3.495994493038772e-02 3.461773231566689e-02 3.496007727466858e-02 3.496013862728978e-02 3.496019616952864e-02 3.453419647204347e-02 3.495994493038771e-02 3.461773231566698e-02 3.496007727466857e-02 3.496013862728979e-02 3.496019616952864e-02 3.453419647204343e-02 3.461773231566688e-02 3.496007727559594e-02 3.496013862728978e-02 3.496019616952863e-02 3.495994493038770e-02 3.461773231566689e-02 3.496007721824784e-02 3.496013862728979e-02 3.496019616952864e-02 3.495994489860085e-02 3.496013862728979e-02 3.495994485406865e-02 3.461773231104990e-02 3.496007753154848e-02
Table 4: and as function of and () - Problem 2
5.773468954018757e-02 5.773118330947371e-02 5.772450392218931e-02 5.771827398570289e-02 5.771249955261612e-02 5.770718624499813e-02 5.773468954018757e-02 5.773118330947368e-02 5.772450392218937e-02 5.771827398570288e-02 5.771249955261611e-02 5.770718624499812e-02 5.773468954018758e-02 5.773118330947370e-02 5.771249955261611e-02 5.770718624499811e-02 5.773468954018757e-02 5.773118330947373e-02 5.772450392218934e-02 5.771827398570289e-02 5.771249955261611e-02 5.770718629444516e-02 5.773468954018760e-02 5.773118330947369e-02 5.772450392218934e-02 5.771827398570289e-02 5.771249952331193e-02 5.770718624499811e-02 5.773118330947372e-02 5.772450392378905e-02 5.771827419320513e-02
5.788632535497507e-02 5.787373829705302e-02 5.784950155085092e-02 5.782658849902559e-02 5.780508422056688e-02 5.778506929509217e-02 5.788632535497507e-02 5.787373829705301e-02 5.784950155085092e-02 5.782658849902555e-02 5.780508422056686e-02 5.778506929509220e-02 5.788632535497510e-02 5.787373829705302e-02 5.784950155085092e-02 5.780508422056686e-02 5.778506929509220e-02 5.788632536157578e-02 5.787373829705302e-02 5.784950155085091e-02 5.782658848730357e-02 5.780506330666032e-02 5.778506929813398e-02 5.788632527723454e-02 5.784950155085083e-02 5.782658849902568e-02 5.788632541333875e-02 5.787373829803212e-02 5.784950155085090e-02 5.782658849917136e-02
5.809599803382869e-02 5.807549989275792e-02 5.803325570359528e-02 5.798937823866770e-02 5.794647454306275e-02 5.790575933289238e-02 5.809599803382870e-02 5.807549989275791e-02 5.803325570359529e-02 5.798937823866770e-02 5.794647454306275e-02 5.790575933289238e-02 5.809599803382870e-02 5.807549989275791e-02 5.803325570359529e-02 5.798937823866768e-02 5.790575933289237e-02 5.809599803382870e-02 5.807549990183274e-02 5.798937823866769e-02 5.794647454306268e-02 5.790575933289235e-02 5.809599807145778e-02 5.807550000874059e-02 5.803325568836281e-02 5.794647454306276e-02 5.790575939801962e-02 5.809599805160266e-02 5.807549983114331e-02 5.803325538684955e-02 5.794647445626924e-02 5.790575931600109e-02
5.830785511535787e-02 5.827839030475356e-02 5.822099874390471e-02 5.816499040667799e-02 5.810964168493156e-02 5.805455701855686e-02 5.830785511535792e-02 5.827839030475357e-02 5.822099874390501e-02 5.816499040667800e-02 5.810964168493161e-02 5.805455701855686e-02 5.830785511535794e-02 5.822099874390471e-02 5.816499040727242e-02 5.810964168493156e-02 5.805455701855683e-02 5.827839030475357e-02 5.822099874390473e-02 5.816499039611871e-02 5.810964168493157e-02 5.805455701855686e-02 5.827839030296449e-02 5.810964168493159e-02 5.827839029079828e-02 5.822099874092245e-02 5.816499015380254e-02
Table 5: and