Log In Sign Up

Statistical harmonization and uncertainty assessment in the comparison of satellite and radiosonde climate variables

by   Francesco Finazzi, et al.

Satellite product validation is key to ensure the delivery of quality products for climate and weather applications. To do this, a fundamental step is the comparison with other instruments, such us radiosonde. This is specially true for Essential Climate Variables such as temperature and humidity. Thanks to a functional data representation, this paper uses a likelihood based approach which exploits the measurement uncertainties in a natural way. In particular the comparison of temperature and humdity radiosonde measurements collected within RAOB network and the corresponding atmospheric profiles derived from IASI interferometers aboard of Metop-A and Metop-B satellites is developed with the aim of understanding the vertical smoothing mismatch uncertainty. Moreover, conventional RAOB functional data representation is assessed by means of a comparison with radiosonde reference measurements given by GRUAN network, which provides high resolution fully traceable radiosouding profiles. In this way the uncertainty related to coarse vertical resolution, or sparseness, of conventional RAOB is assessed. It has been found that the uncertainty of vertical smoothing mismatch averaged along the profile is 0.50 K for temperature and 0.16 g/kg for water vapour mixing ratio. Moreover the uncertainty related to RAOB sparseness, averaged along the profile is 0.29 K for temperature and 0.13 g/kg for water vapour mixing ratio.


page 1

page 2

page 3

page 4


AODisaggregation: toward global aerosol vertical profiles

Aerosol-cloud interactions constitute the largest source of uncertainty ...

Coupling Oceanic Observation Systems to Study Mesoscale Ocean Dynamics

Understanding local currents in the North Atlantic region of the ocean i...

Integrated monitoring of ice in selected Swiss lakes. Final project report

Various lake observables, including lake ice, are related to climate and...

Calibrated and Enhanced NRLMSIS 2.0 Model with Uncertainty Quantification

The Mass Spectrometer and Incoherent Scatter radar (MSIS) model family h...

Analyzing High-Resolution Clouds and Convection using Multi-Channel VAEs

Understanding the details of small-scale convection and storm formation ...

A Hierarchy of Empirical Models of Plasma Profiles and Transport

Two families of statistical models are presented which generalize global...

Bayesian Assessments of Aeroengine Performance

Aeroengine performance is determined by temperature and pressure profile...

1 Introduction

Satellite validation is key to ensure that satellite products meet the mission specified requirements for climate and weather applications. Since the agreement of satellite measurements with ground-based reference measurements is an essential quality indicator, one major issue in performing a rigorous validation is the quantification of the uncertainty due to the co-location mismatch in time and space between satellite and ground based reference observations. This mismatch is due to the different sampling of atmosphere carried out by the two instruments (Verhoelst et al., 2015), which are also quite often based on very different sensing techniques. Moreover, satellite and ground based observations are typically collected on very different time scales and spatial scales. As a consequence, in a satellite vs ground measurements comparison, we may have horizontal and/or vertical and/or temporal mismatches in smoothing and/or profile resolution, which contribute to the overall co-location mismatch uncertainty (hereinafter mismatch uncertainty).

Over the last decade, several authors tried to estimate the impact of the satellite vs. ground based mismatch uncertainty of the Essential Climate Variables (ECVs). The most common approach is to use simple descriptive statistics to identify the maximum temporal and spatial distances which warranty a controlled mismatch uncertainty. See e.g. Pappalardo et al. (2010) for aerosols and Kursinski & Hajj (2001) for water vapor. Moreover, considering vertical resolution, Pougatchev et al. (2007) found that, when available, the averaging kernels can be used to reconcile the vertical resolution of satellite and ground based observations of temperature and humidity.

A rigorous metrological characterization of the mismatch uncertainty requires the quantification of the total uncertainty budget for each satellite retrieved ECV. Hence, the uncertainty budget includes the contribution of random, systematic, sampling, smoothing uncertainties and their correlation with all the relevant environmental factors. Pioneering works in this direction are Ridolfi et al. and Lambert et al. . More recently, Verhoelst et al. (2015) used an explicit physic simulation method for computing a full uncertainty budget closure for ozone. Moreover, Fassò et al. (2014) and Ignaccolo et al. proposed an approach based on the extension of the classical functional regression model able to cover for hetereoskedasticity of mismatch error in temperature and humidity observation.

GAIA-CLIM project ( is a Horizon 2020 project which aims at improving the use of non-satellite measurements to characterise, calibrate and validate satellite measurements. Considering temperature and humidity, one of its objectives is to understand the mismatch uncertainty in the comparison of the satellite observations obtained by the Infrared Atmospheric Sounding Interferometer (IASI) instrument, on board EUMETSAT MetOP-A and –B, with the radiosonde observations (RAOBs). In fact RAOB profiles are appealing for satellite validation because of their extensive spatial and temporal coverage, hence permitting the assessment of mismatch uncertainty at global level. Despite of this, RAOB observations cannot be strictly considered reference measurements because they are not fully traceable and have a limited vertical resolution, see Dirksen et al. . Note that the GCOS Reference Upper-Air Network (GRUAN, provides reference products, which are fully traceable but have a very limited spatial coverage (16 sites as the paper is written).

Along these lines, the present paper focuses on the vertical smoothing mismatch uncertainty of IASI-RAOB profile comparison of temperature and humidity. This objective is achieved by a statistical technique for vertical harmonization which is independent on the availability of IASI averaging kernels, hence especially relevant for comparisons where averaging kernels are not available, in particular for historical data analysis. To do this, the vertical data-point sparseness of RAOB network is assessed by means of a comparisons with GRUAN reference products where available.

The proposed technique is a two steps technique. At the first step RAOB profiles are transformed into continuous functions using splines, which are optimized to match as close as possible to GRUAN profiles. In doing this, vertical sparseness uncertainty and processing mismatch uncertainty are assessed. At the second step RAOB profiles are harmonized by considering weighting functions based on the Generalized Extreme Values (GEV) probability density function (pdf) whose parameters depend on the IASI levels.

The paper is structured as follows. In Section 2 data from both satellite (IASI) and radiosonde (GRUAN and RAOB) are introduced. In Section 3, the various sources of uncertainty arising in satellite-ground comparisons are reviewed. Sections 4 and 5 discuss novel statistical modeling: the former section leverages on intuition while the latter embeds the same model in a rigorous maximum likelihood estimation problem. Section 6 applies this approach to IASI-RAOB comparison for a number of RAOB stations in central Europe. To do this, in Section 6 the RAOB soundings are transformed into functional data and harmonized to match IASI vertical smoothing. Then the sparseness uncertainty of RAOB and vertical smoothing uncertainty of IASI-RAOB comparison are computed. Section 7 gives concluding remarks.

2 Data sets

The data sets used in this study include atmospheric profile retrievals derived from IASI instrument and from conventional (RAOB) and reference (GRUAN) radiosonde networks. The RAOB-IASI co-location data set, which is provided by NOAA-NESDIS, has been collected through the NOAA Products Validation System (NPROVS), see Reale et al. and Sun et al. ( The NPROVS data set used in this study includes co-located profiles at 21 RAOB stations selected across the central European area (C-EU), described in Figure 1, for the period January 2015 – February 2016.

Figure 1: Spatial distribution of RAOB stations (red circles) in Central Europe and RAOB-GRUAN station in Lindenberg (cyan star). The number near each dot is the number of launches during the period January 2015 - February 2016.

Each co-location pair includes RAOB and IASI profiles for temperature and water vapor mixing ratio (WVMR) with RAOB at mandatory and significant levels and IASI at 100 levels. In order to consider profiles with enough data, the data set has been filtered as follows: temperature (WVMR) has been restricted to atmospheric range ( ) and only co-locations with at least RAOB measurements have been selected for the analysis, giving () co-locations out of the original NPROVS co-locations. Notice that the atmospheric range considered after filtering is still relevant for climate and weather studies as pressure levels and , correspond to around and respectively.

In addition, the GRUAN station at Lindenberg, has been used as a reference for radiosounding measurements to understand conventional RAOB sparseness uncertainty. In fact this GRUAN station is also a conventional RAOB station and, although the instrument is physically the same for both, data are processed in a different manner giving different measurements. As a result for temperature (WVMR) we have () GRUAN-RAOB co-locations.

Other meteorological variables considered in statistical modelling of the mismatch error, like wind, solar radiation or geopotential, have been taken from the ERA-Interim global atmospheric reanalysis implemented by the European Centre for Medium-Range Weather Forecasts (ECMWF), see Berrisford et al. (2011).

2.1 Iasi

Products retrieved from EUMETSAT’s IASI instruments aboard of MetOp-A and Met-Op B satellites considered in this study are based on version 6 of the IASI level 2 processor. The infrared atmospheric sounding interferometer (IASI) is a Fourier transform spectrometer based on the Michelson interferometer, associated with an integrated imaging system (Blumstein et al., 2004).

The IASI atmospheric profiles of temperature (WVMR) are available at 74 pressure levels in the range ( ). Considering vertical smoothing, the IASI sounding products represent thermodynamic states of deep atmospheric layers at variable depths, due to the integrating nature of the radiation measurements at the top of the atmosphere. The maximum number of independent pieces of information is approximately 14 (10) for temperature (humidity) profiles, the exact number depending on atmospheric conditions. Hence the true vertical resolution is quite lower than the vertical grid of 74 pressure levels discussed above and profiles retrieved from such radiance measurements are smoothed versions, where the smoothing functions are given by the so-called averaging kernels. Although version 6 of the IASI level 2 processor provides the information to calculate the averaging kernels, it has not been used in this paper, being not available in the NPROVS data set.

2.2 Conventional RAOB

Conventional radiosonde observations have been used historically as a de facto standard data set in satellite calibration (via radiative transfer models) and validation. Worldwide there are more than 2000 radiosonde launch sites and mobile ship-based launch station.

While a radiosonde transmits an essentially continuous stream of temperature and humidity information back to the station (each 5-10 m of altitude, measured each 1-2 s), for temperature (WVMR), RAOB data set includes only 15 (6) "mandatory levels" in the atmospheric range ( ). Moreover data are given at various "significant levels" which are the pressure levels where a significant change or an extreme is identified in the vertical temperature and/or dewpoint temperature profiles. For this reason, the ECV variation between two such significant level is often assumed close to linear. In practice, altitude and number of significant levels change among different profiles and, on average, 28 significant levels per profile are available in the RAOBs collected at NPROVS, the exact numbers depending on specific atmospheric conditions.

2.3 Gruan

Conventional RAOBs may not be able to provide reference-quality in situ and ground-based remote sensing observations of upper-air essential climate variables for metrological and traceability reasons, see Seidel et al. , Immler et al. and Bojinski et al. . Improving on this, GRUAN data processing was developed to meet the criteria for reference measurements (Dirksen et al., 2014). As a result, GRUAN radiosounding profiles are provided together with individual measurement uncertainty estimates at high vertical/temporal resolution: measurements are obtained at 1-2 s or 5-10 m in altitude; this temporal resolution is then reduced to about 10 s during processing by a low pass filter to avoid temperature spikes. GRUAN quality has been extensively assessed, see e.g. Calbet et al. . Since Lindenberg GRUAN station is also a conventional RAOB station, it is important to remark here that, considering this station, the two profiles differ for the vertical resolution and for data processing. In fact the former is obtained using the GRUAN processing algorithm while the latter is obtained using the algorithm implemented in Vaisala RS92 instruments. As a result, the two products give non coinciding measurements.

3 Co-location mismatch sources

As above discussed, the comparison of radiosonde and IASI profiles aims at understanding which factors contribute to the discrepancies observed between a satellite vertical profile and a comparator profile. In principle the comparator is taken as an error free "true" state, but, in practice, its uncertainty is worth to be considered.

In fact a meaningful comparison should take into account: the spatio-temporal mismatch between profiles; the different vertical smoothing and resolution of the two instruments/data sets; the different horizontal resolution of the two instruments. Since the latter is not important for IASI which has a relatively small footprint, the following subsections briefly discuss the former points and call for a comparison based on data harmonization.

3.1 Satellite vertical smoothing

As discussed in Section 2, radiosonde and IASI are based on completely different measurement techniques. While the radiosonde is able to make a "direct" measurement of the ECV at the position in space and time reached by the weather balloon, IASI sounds the atmosphere using an interferometry technique. This implies that the vertical resolution of IASI is much lower than the resolution characterizing a radiosonde. Any comparison between radiosonde and IASI profiles, thus, may be affected by these differences.

Note that different methods are available in satellite product validation to resolve the issue in vertical resolution difference. One requires to apply satellite sounder averaging kernel to the target data, e.g., radiosonde data, and then to compare with the retrievals (Maddy et al. 2008 ); one is to first average vertical layer for both satellite and target data profiles, and then to compute the validation statistics at those “coarse” vertical layers (Tobin et al. 2006; Sun et al. 2017). In this study, we choose to employ the vertical harmonization technique for the uncertainty assessment, see Section 4.

3.2 Comparator uncertainty and vertical sparseness

It has been seen that RAOB data are at sub-reference level and are provided at the so-called mandatory and significant pressure levels, which are sparse vertically. The latter being given at pressure levels where some interesting variation is happening. This entails that data occurs according to a preferential sampling design (Diggle et al., ) which is dependent on second order derivatives.

As a consequence, before developing a IASI comparison, the estimation of the "true" profile at any pressure level, based on RAOB data, requires a statistical assessment. In this frame, (vertical) sparseness uncertainty is the uncertainty component related to the coarse vertical resolution.

3.3 Spatio-temporal mismatch

Radiosonde and IASI profiles are characterized by a spatio-temporal mismatch. For the data set considered, only co-locations with horizontal distances up to km at surface and time delays up to three hours have been considered, see Kursinski and Hajj . This is because it is impossible to perfectly synchronize the weather balloon launch with the satellite overpass, and the overpass may be far from the station where the radiosonde is launched, both in space and time. Additionally, the IASI profile is retrieved nearly instantly while weather balloons take on average 1.7 hrs from surface up to 10 hPa (Seidel at el. 2011). Moreover, the latter is shifted by winds during its ascent. This means that profiles are never perfectly co-located even if the satellite nadir viewing overpasses the launch station.

4 Statistical harmonization and uncertainties

Vertical harmonization refers to a data transformation, which reduces the differences in the vertical smoothing between the two profiles and improves the radiosonde and IASI profiles comparability. In our case, the low vertical resolution of IASI implies that IASI retrievals are much smoother than radiosonde data. Since we cannot un-smooth IASI profiles, the radiosonde profiles are smoothed in order to mimic the IASI retrievals.

This is given by the convolution of the radiosonde profile of temperature and humidity with a normalized weighting function in the pressure range . Namely



The "true" profile may be assumed a continuous function of pressure, but RAOB profiles are observed only at a limited number of "preferential" levels. To handle this a two steps procedure is proposed. The first step, developed in Section 4.2, extends the idea of Fassò et al. to represent atmospheric profiles as functional data (Ramsay and Silverman, ) with smoothness coefficient obtained by minimizing the difference with the reference GRAUN data. The second step, described in Sections 4.3 and 4.4, optimizes the weighting function .

4.1 Data and likelihood function

Let us consider a collection of co-located RAOB-IASI profiles observed across the geographic area and time frame of interest, with a subset of these co-locations, say obtained at Lindenberg station, and having also GRUAN profile counterparts.

For each given co-location , let

be the radiosonde data vector related to pressure levels

, with for RAOB or for GRUAN. Moreover let be the IASI data vector related to pressure levels . Note that the RAOB pressure levels depends on co-location . Instead IASI pressure levels are invariant among co-locations with (32) for temperature (WVMR). As a consequence RAOB and IASI pressure levels are different and vertical matching may represent an issue. On the contrary, the number of GRUAN measurements is very high for all profiles, so that for any prefixed RAOB level in a very close GRUAN pressure level in may be found.

4.2 RAOB estimation and sparseness uncertainty

The discrepancies between conventional RAOB and GRUAN may be used to understand the loss of information of RAOB due to its sparse vertical resolution. In fact, the minimization of this loss can be used to define an optimal estimate of the unobserved true signal.

To see this, the true signal of profile is considered as a smooth function denoted by , and it is related to observation for and by the following conditions



is Gaussian distributed,

, and


These three conditions will be discussed in details later. For the moment note that

is a smooth bias, constant over co-locations and is a weighting function. Moreover, note that, for GRUAN, the squared measurement uncertainty is known at all pressure levels and . For RAOB the measurement uncertainty is not widely available but there is some evidence that for some . For simplicity we assume tha does not depend on pressure level nor on co-location .

In this paper, we estimate the smooth profile by which is a penalized spline with smoothing factor . The estimated profile is computed on RAOB data by solving the following penalized weighted least square problem:




for co-locations in Lindeberg and else.

Following e.g. Reinsch , and using one knot per observation, the solution of equation may be expressed in terms of tolerance , which is the upper limit of weighted root mean squared error along the RAOB profile


where, clearly,

gives interpolating splines. For this reason, depending on the context, we will use either

or to address smoothing properties of spline .

Using GRUAN-RAOB comparison at Lindenberg, we estimate the bias by the weighted RAOB-GRUAN average difference, namely


Next, the smoothing factor is obtained by optimizing the adjusted GRUAN-RAOB difference. In other words, is the solution of the following weighted least squares criterion:


where is defined in Equation .

Due to the peculiarity of the RAOB sampling points discussed in Section 2.2, three spline models are compared in Section 6: linear and cubic smoothing Bsplines and Hermite interpolating splines (Hsplines). The former two are well known and we only remark here that the smoothing coefficient is not obtained by a cross-validation or generalized CV criterion on RAOB data as in standard smoothing splines. Instead is numerically optimized according to the GRUAN agreement criterion which takes into account measurement uncertainty. Interpolating Hsplines are also known as piecewise cubic Hermite interpolating polynomials, being cubic monotonic splines with continuous first derivatives, see Fritsch and Carlson  Hence Hsplines are introduced here as an intermediate solution between cubic and linear Bsplines. In fact, in this model selection problem, one could use also smoothing Hsplines, this approach being further discussed in the case study.

After obtaining , the optimized quantity in Equation provides the total mismatch uncertainty profile of RAOB-GRUAN comparison, namely:


which is the loss of information due to sparseness and difference in data processing. More comments and the decomposition of Equation 10 in sparseness and processing uncertainty will be developd in the case study.

4.3 Vertical smoothing

With the aim of making radiosonde and IASI profiles comparable, the radiosonde profile of the previous section, denoted in the sequel by is smoothed by means of a weighted integral, namely:


In Equation , the weight function is a non negative and normalized weight function, depending on and a parameter vector , which has to be estimated. Since the role of is to mimic the IASI sounding of the atmosphere, Fassò et al. considered the following alternative functions: rectangular, sine, Gaussian and GEV distribution. As expected the latter was found to outperform the other simpler competitors and, for this reason, it is used here. In particular the GEV pdf has parameter vector , which are the location, scale and shape parameter, respectively, see Kotz and Nadarajah In this paper we use level dependent parameters, namely and .

In order to compute the harmonized RAOB from Equation 11, we need to estimate and a natural choice is the following penalized weighted least squares iterated for


where and the weight is the normalized squared reciprocal measurement uncertainty of IASI, analogous to formula . Moreover where

is a variance covariance matrix to be discussed in the next section and

if and else. Note that the penalty term in (12) is related to smoothness of the atmosphere and hence of wrt .

4.4 Vertical smoothing uncertainty

A byproduct of the data harmonization procedure above described is the uncertainty component related to the vertical smoothing. In particular, the RAOB-IASI mismatch uncertainty due to difference in vertical smoothing is given by


where is the vertically harmonized mismatch uncertainty:


and is the raw mismatch uncertainty in the comparison of IASI with non harmonized RAOB namely

Hence, may be interpreted as the (squared) mismatch uncertainty improved by the data harmonization.

5 Likelihood inference

The modeling machinery of the previous section has a rigorous interpretation as a maximum likelihood estimation problem for a non linear mixed effect model. This is properly described using three main steps for RAOB, GRUAN and IASI respectively.

5.1 RAOB likelihood

The first step is to represent the RAOB true signal by the following linear representation


where is the vector of Bspline basis functions and is the vector of the spline coefficients. Using this, Equation for RAOB profile may be rewritten as follows:

where .

Stacking in a vector, say , all in a matrix and all errors in a vector we have the following matrix representation

This has a well known mixed effects model interpretation (see e.g. Fahrmeir et al., ). To see this, let

denotes the probability distribution of the random vector

and assume that, where , is a known smoothing factor and is the identity matrix. Than the penalized least square criterion in Equation corresponds to the maximum likelihood estimate (MLE). In fact, apart for an additive constant, we have

Stacking all in a vector , and similarly for and , we have the following matrix representation


where is diagonal matrix corresponding to uncertainties in Equation 6. Since this likelihood is optimized by minimizing each summand independently, the computation burden is linear in and the solution is the MLE as a function of (or ).

Hence Equation may be rewritten as

Notice that the RAOB data model for may be partitioned as

where is the GRUAN matching data set corresponding to co-locations , and is the remaining major part of the RAOB data set with soundings.

5.2 GRUAN-RAOB likelihood

Now, considering GRUAN and RAOB matching data in , we have the following representation

where is a GRUAN-RAOB fixed effect bias. Hence, in matrix notation, we may write


If we compute this at we have a profile log-likelihood which is easily optimized for and finally for We than have the MLE triplet for IASI and RAOB data sets:

which is given by Equations and .

5.3 IASI-RAOB likelihood

The IASI observation equation is obtained by substituting Equations and in Equation This gives


where and .

Hence the stacked IASI observation equation for pressure level and all co-locations may be written as follows:

and the full data set is represented by where .

Now, in order to estimate , one could consider independent estimates for separately. But this assumption contrasts with atmospheric considerations and tends to overfit. On the opposite side, one could assume is a smooth function of and use Bspline. This would largely increase the number of parameters to be simultaneously optimized, resulting in an unfeasible algorithm. An intermediate and suitable solution is to assume that is a vector random walk, namely


for In the equation above, is an unknown parameter and the innovations are Gaussian distributed with a diagonal matrix.

It follows that, the profile likelihood for , and known and is given by


which is optimized by the following nonlinear optimizations


for , which correspond to minimizations in Equation . Note that the diagonal matrix is given by the uncertainties in Equation which are assumed to be known up to an acceptable approximation.

6 Case study

In this section, the two step harmonization procedure presented in Section 4 is applied to the RAOB-IASI data set introduced in Section 2, independently for temperature and WVMR co-locations.

6.1 RAOB estimation

The first step is the transformation of the sparse RAOB radiosonde profiles into continuous functions to be used in the convolution of step two. To do this, spline type and smoothing level have been chosen according to the GRUAN closeness criterion of Equation . Considering linear Bsplines, smoothing optimization is shown in Figures 2 and giving and for temperature and humidity respectively. Table 1 shows that linear Bsplines with GRUAN-optimal smoothing improves over both cubic Bsplines and interpolating Hsplines. This is consistent with the preferential sampling design characterizing RAOB significant levels mentioned in Section 4.2.

Temperature WVMR
Linear Bsplines 0.4897 0.0346
Cubic Bsplines 0.7706 0.0712
Interpolating Hsplines 0.6085 0.0379
Table 1: Comparison of linear and cubic Bsplines and interpolating Hermite splines (Hsplines) of temperature (K) and humidity (g/kg), based on weighted root mean square error of RAOB-GRUAN in Lindenberg.
Figure 2: Temperature. Smoothing optimization of RAOB linear Bsplines wrt GRUAN data in Lindenberg. Abscissa: tolerance given by Equation . Ordinate: RAOB-GRUAN weighted root mean square error.
Figure 3: WVMR. Smoothing optimization of RAOB linear Bsplines wrt GRUAN data in Lindenberg. Abscissa: tolerance given by Equation . Ordinate: RAOB-GRUAN weighted root mean square error.

6.2 Sparseness and processing uncertainty

The comparison of RAOB and GRUAN data in Lindenberg station provides the GRUAN-RAOB total mismatch uncertainty, which is computed using the approach of Section 4.2. In particular in Figures 4 and 5, of Equation shows a peculiar behavior with local minima at mandatory levels. In fact, as discussed in Section 2, both RAOB and GRUAN are observed at these pressure levels, while between them, RAOB is observed only at significant levels. The red line interpolates between above minima and defines the mismatch due to difference between GRUAN data processing (Dirksen et al, 2014) and Vaisala RS92 data processing, denoted by . As a result the black dashed line of Figures 4 and 5 is the sparseness uncertainty adjusted for mismatch in processing and is given by the quadratic difference among the previous uncertainties, namely

Figure 4: Temperature. RAOB-GRUAN mismatch uncertainties: solid blu line is total mismatch uncertainty ; solid red line is uncertainty due to difference between Vaisala and GRUAN processing ; dashed black line is the RAOB sparseness uncertainty .
Figure 5: WVMR. RAOB-GRUAN mismatch uncertainties: solid blu line is total mismatch uncertainty ; solid red line is uncertainty due to difference between Vaisala and GRUAN processing ; dashed black line is the RAOB sparseness uncertainty .

Considering temperature the processing uncertainty is close to until . In this range also sparseness uncertainty is generally smaller than . In the upper atmosphere both uncertainties are larger consistently with solar radiation bias. Considering WVMR, as expected, the vertical pattern is reversed with a processing uncertainty decreasing from at ground level to at In this range the sparseness uncertainty is smaller

6.3 Harmonization and vertical smoothing

Conventional RAOB profiles are harmonized thanks to the optimization in Equation This is solved for each IASI pressure level iteratively from the top pressure level and going down to separately for temperature and WVMR. We tried also to iterate in the opposite order, from ground to upper air, obtaining very close results. At each pressure level the optimization is solved numerically. Since it is reasonable to assume that nearby pressure levels are characterized by a similar , the initial value for is set to zero for all . To avoid local minima, the optimization for is repeated times with randomly perturbed initial values and is taken as the optimum of these solutions. The diagonal variance covariance matrix , which acts as a smoothing factor has been obtained by a preliminary not regularized estimation run.

In order to illustrate the results, Figures 6 and 7 show GEV pdf’s related to IASI pressure levels for temperature and WVMR. Each function essentially mimics how IASI sounds the atmosphere at each specific pressure level with a peak near the corresponding IASI level. The smaller the function width at pressure level , the better the IASI retrieval describes the ECV at that level.

Note that, in Figure 6, the pdf dispersion tends to decrease at upper altitudes, especially above . This is mainly due to the non linearity of the pressure scale. For instance, a pressure difference of at corresponds to an altitude difference of around , while the same pressure difference at corresponds to an altitude difference of only . Moreover in Figure 7 weight functions near are clearly affected by a border effect and should be interpreted with caution.

Figure 6: Temperature. Weight functions given by GEV pdf’s for IASI pressure levels.
Figure 7: WVMR. Weight functions given by GEV pdf’s for IASI pressure levels.

After harmonizing RAOB to IASI, the adjusted uncertainty of Equation is computed and the related vertical smoothing uncertainty decomposition of Equation , is reported in Figures 8 and 9. The IASI-RAOB comparison is dominated by the smoothness of the IASI retrieval and its reduced capability to catch strong vertical gradients with respect to the RAOB profiles, though their sparseness. In the boundary layer (BL) below about 900 hPa, where significant inversion in the temperature profiles may occur, the hamonization does not strongly reduce the raw uncertainty but above, up to 700 hPa, the reduction becomes more significant. In the upper troposphere/lower stratosphere the strong gradients at the tropopause increase the raw uncertainty and the harmonization strongly reduces the difference between IASI and RAOB. It is worth to remind that the values calculated above 100 hPa are affected by the size of the sampling which is more limited that at higher pressure levels. For WVMR, results similar to temperature are observed in the BL. The increase of the smoothness uncertainty at 800 hPa is likely linked to the transition from wetter to drier air occurring at the top of BL not always caught in the RAOB data. The benefit of the harmonization decreases with the altitude proportionally with the decrease of the water vapour variability in the atmosphere. This is clearly visible from the difference between the raw and harmonized uncertainties which reduces with the height.

Figure 8: Temperature. Vertical smoothing uncertainty: , black dotted line; it is the uncertainty due to difference in smoothing between RAOB and IASI, profile average is . Harmonized mismatch uncertainty: , cyan dashed line; it is the uncertainty due to mismatch after adjusting for difference in vertical smoothing, profile average is . Unadjusted uncertainty: , red solid line; it is the total uncertainty between interpolated RAOB and IASI, profile average is . Formulas given in Section 4.4.
Figure 9: WVMR. Vertical smoothing uncertainty: , black dotted line; it is the uncertainty due to difference in smoothing between RAOB and IASI, profile average is . Harmonized mismatch uncertainty: , cyan dashed line; it is the uncertainty due to mismatch after adjusting for difference in vertical smoothing, profile average is . Unadjusted uncertainty: , red solid line; it is the raw uncertainty between interpolated RAOB and IASI, profile average is . Formulas given in Section 4.4.

7 Discussion and conclusion

This paper discussed the comparison of IASI and RAOB temperature and humidity with a focus on vertical smoothing. Since the IASI averaging kernels have been considered unknown, a weighting function mimicking the weights of the averaging kernel has been estimated on data. To do this RAOB data have been transformed in functional data and the related uncertainty has been assessed by a comparison with the reference measurements for radiosonde given by GRUAN data, Lindenberg. Hence it can be considered as a first substantial step in the direction of Calbet et al. 2017 "To fully characterize the comparison, a method to estimate the collocation uncertainty would be desirable. This method should not depend on the data being used for the study and should be independent from them".

Thanks to this approach, it has been found that the uncertainty of vertical smoothing mismatch averaged along the profile is for temperature and g/kg for water vapour mixing ratio. Moreover, the uncertainty related to RAOB vertical sparseness, averaged along the profile is for temperature and g/kg for water vapour mixing ratio.

From the methodological point of view, it has been shown that the estimates are obtained by the maximum likelihood method, taking into consideration also the measurement uncertainties where available.

7.1 Further developments

Further aspects may be added to the presente analysis of satellite vs radiosonde comparison. For instance, the distance between the satellite line of sight and the radiosonde position has not been considered. In fact this issue will be addressed in a forthcoming paper using isotonic regression (Mayer, ).

Vertical correlation has not been considered explicitly here and, in this sense, the results could be suboptimal. In fact, IASI measurements are known to have a limited number of degrees of freedom. In our approach at least a part of IASI vertical correlation is implicitly handled by the random walk dynamics of GEV pdf parameters in Section


. Considering radiosonde, sources of vertical correlation arise both from short range smoothing algorithms, used to avoid measurement outliers, and by pre-launch calibration errors. Although a good part of these problems is automatically handled by the functional data approach used here, further research could point out new solutions.

A further insight into vertical smoothing could benefit from the comparison of this proposal with the IASI "true averaging kernel" at least in some cases. Nonetheless, we remark that the approach of this paper can be used even in absence of averaging kernels, which is quite relevant especially for historical records.


This research is partially funded by GAIA-CLIM, the project funded from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 640276.

We are very thankful to NOAA and in particular to Tony Reale for providing the radiosonde-satellite collocations.


  • [1] Berrisford P, Dee DP, Fielding K, Fuentes M, Kållberg P, Kobayashi S, Uppala SM. ‘The ERA-Interim Archive’. ERA Report Series, No. 1. ECMWF: Reading, UK.
  • [2] Bojinski, S., Verstraete, M., Peterson, T. C., Richter, C., Simmons, A., and Zemp, M. The concept of Essential Climate Variables in support of climate research, applications, and policy, B. Am. Me- teorol. Soc., 95, 1431–1443, doi:10.1175/bams-d-13-00047.1.
  • [3] Calbet, X., Peinado-Galan, N., Ripodas, P., Trent, T., Dirksen, R., and Sommer, M. Consistency between GRUAN sondes, LBLRTM and IASI. Atmos. Meas. Tech., 10(6), 2323-2335.
  • [4] Dirksen, R. J., Sommer, M., Immler, F. J., Hurst, D. F., Kivi, R., and Vömel, H. Reference quality upper-air measurements: GRUAN data processing for the Vaisala RS92 radiosonde. Atmos. Meas. Tech., 7, 4463-4490,
  • [5] Fahrmeir L., Kneib T., Lang S. & Marx B. Regression - Models, Methods and Applications. Springer. Berlin.
  • [6] Fassò, A., Ignaccolo, R., Madonna, F., Demoz, B. and Franco-Villoria M. Statistical modelling of collocation uncertainty in atmospheric thermodynamic profiles. Atmos. Meas. Tech. 7, 1803–1816.
  • [7] Ignaccolo, R., Franco-Villoria, M., Fassò, A. Modelling collocation uncertainty of 3D atmospheric profiles. Stochastic Environmental Research and Risk Assessment 29 (2), 417-429.
  • [8] Immler, F. J., Dykema, J., Gardiner, T., Whiteman, D. N., Thorne, P. W., and Vömel, H. Reference Quality Upper-Air Measurements: guidance for developing GRUAN data products, Atmos. Meas. Tech., 3, 1217–1231, doi:10.5194/amt-3-1217-2010.
  • [9] Kotz, S., Nadarajah S. Extreme Value Distributions: Theory and Applications. Imperial College Press, London, UK.
  • [10] Kursinski, E. R., and G. A. Hajj, A comparison of water vapor derived from GPS occultations and global weather analyses, J. Geophys. Res., 106, 1113–1138.
  • [11] Lambert, J.C. and Vandenbussche, S. Multi-dimensional characterisation of remotely sensed data. EC FP6 GEOmon Technical Note D4.2.1. GEOmon TN-IASB-OBSOP, BIRA-IASB.
  • [12] Meyer, M.C. A Simple New Algorithm for Quadratic Programming with Applications in Statistics, Communications in Statistics, 42(5), 1126-1139.
  • [13] Pappalardo, G., Wandinger, U., Lucia, M., Hiebsch, A. Mattis, I., et al. (2010), EARLINET correlative measurements for CALIPSO: First intercomparison results, J. Geophys. Res., 115, D00H19, doi:10.1029/2009JD012147.
  • [14] Pougatchev, N., August, T., Calbet, X., Hultberg, T., Oduleye, O., Schlüssel, P., Stiller, B., Germain, K. St., and Bingham, G. IASI temperature and water vapor retrievals – error assessment and validation, Atmos. Chem. Phys., 9, 6453–6458, doi:10.5194/acp- 9-6453-2009.
  • [15] Ravegnani, F., Reburn, W. J., Redaelli, G., Remedios, J. J., Sembhi, H., Smale, D., Steck, T., Taddei, A., Varotsos, C., Vigouroux, C., Waterfall, A., Wet- zel, G., and Wood, S. Geophysical validation of MIPAS- ENVISAT operational ozone data, Atmos. Chem. Phys., 7, 4807– 4867, doi:10.5194/acp-7-4807-2007.
  • [16] Ramsay, J., Silverman, B. Applied Functional Data Analysis. Methods and Case Studies. Springer.
  • [17] Reinsch C. "Smoothing by spline functions II", Numerische. Mathematik. 16, 451-454.
  • [18] Ridolfi, M., Blum, U., Carli, B., Catoire, V., Ceccherini, et al. Geophysical validation of temperature retrieved by the ESA processor from MIPAS/ENVISAT atmospheric limb- emission measurements, Atmos. Chem. Phys., 7, 4459–4487, doi:10.5194/acp-7-4459-2007.
  • [19] Seidel, D. J., B. Sun, M. Pettey, and A. Reale , Global radiosonde balloon drift statistics, J. Geophys. Res., 116, D07102, doi:10.1029/2010JD014891
  • [20] Sun, B., Reale, A., Tilley, F., Pettey, M., Nalli, N. R., Barnet, C. D. Assessment of NUCAPS S-NPP CrIS/ATMS sounding products using reference and conventional radiosonde observations. IEEE Journal of Selected Topics in Applied Earth Obsrevations and Remote Sensing. 10:6, pp 2499-2509. DOI: 10.1109/JSTARS.2017.2670504.
  • [21] Tobin, D. C., Revercomb, H. E., Knuteson, R. O., Lesht, B. M., Strow, L. L., Hannon, S. E., Feltz, W. F., Moy, L. A., Fetzer, E. J., and Cress, T. S. Atmospheric Radiation Measurement site at- mospheric state best estimates for Atmospheric Infrared Sounder temperature and water vapor retrieval validation, J. Geophys. Res., 111, D09S14, doi:10.1029/2005JD006103.
  • [22] Verhoelst, T., Granville, J., Hendrick, F., Köhler, U., Lerot, C., Pommereau, J.-P., Redondas, A., Van Roozendael, M., and Lambert, J.-C. Metrology of ground-based satellite validation: co-location mismatch and smoothing issues of total ozone comparisons, Atmos. Meas. Tech., 8, 5039-5062, doi:10.5194/amt-8-5039-2015.
  • [23] Maddy, E.S., and Barnet, C.D. Vertical resolution estimates in version 5 of AIRS operational retrievals. IEEE Transactions on Geoscience and Remote Sensing. 46, pp. 2375-2384.