Integrating pre-processing pipelines in ODC based framework

10/04/2022
by   U. Otamendi, et al.
0

Using on-demand processing pipelines to generate virtual geospatial products is beneficial to optimizing resource management and decreasing processing requirements and data storage space. Additionally, pre-processed products improve data quality for data-driven analytical algorithms, such as machine learning or deep learning models. This paper proposes a method to integrate virtual products based on integrating open-source processing pipelines. In order to validate and evaluate the functioning of this approach, we have integrated it into a geo-imagery management framework based on Open Data Cube (ODC). To validate the methodology, we have performed three experiments developing on-demand processing pipelines using multi-sensor remote sensing data, for instance, Sentinel-1 and Sentinel-2. These pipelines are integrated using open-source processing frameworks.

READ FULL TEXT

page 2

page 3

research
05/09/2022

On Designing Data Models for Energy Feature Stores

The digitization of the energy infrastructure enables new, data driven, ...
research
11/04/2022

Rethinking Storage Management for Data Processing Pipelines in Cloud Data Centers

Data processing frameworks such as Apache Beam and Apache Spark are used...
research
10/04/2022

Geo-imagery management and statistical processing in a regional context using Open Data Cube

We propose a methodology to manage and process remote sensing and geo-im...
research
05/15/2017

Probabilistic Matrix Factorization for Automated Machine Learning

In order to achieve state-of-the-art performance, modern machine learnin...
research
01/12/2023

Improvement of Computational Performance of Evolutionary AutoML in a Heterogeneous Environment

Resource-intensive computations are a major factor that limits the effec...
research
08/20/2021

A Recommender System for Scientific Datasets and Analysis Pipelines

Scientific datasets and analysis pipelines are increasingly being shared...
research
08/03/2023

DaphneSched: A Scheduler for Integrated Data Analysis Pipelines

DAPHNE is a new open-source software infrastructure designed to address ...

Please sign up or login with your details

Forgot password? Click here to reset