DOD-ETL: Distributed On-Demand ETL for Near Real-Time Business Intelligence

07/15/2019
by   Gustavo V. Machado, et al.
0

The competitive dynamics of the globalized market demand information on the internal and external reality of corporations. Information is a precious asset and is responsible for establishing key advantages to enable companies to maintain their leadership. However, reliable, rich information is no longer the only goal. The time frame to extract information from data determines its usefulness. This work proposes DOD-ETL, a tool that addresses, in an innovative manner, the main bottleneck in Business Intelligence solutions, the Extract Transform Load process (ETL), providing it in near real-time. DODETL achieves this by combining an on-demand data stream pipeline with a distributed, parallel and technology-independent architecture with in-memory caching and efficient data partitioning. We compared DOD-ETL with other Stream Processing frameworks used to perform near real-time ETL and found DOD-ETL executes workloads up to 10 times faster. We have deployed it in a large steelworks as a replacement for its previous ETL solution, enabling near real-time reports previously unavailable.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/16/2022

Redis for Market Monitoring

This note shows how to use Redis cache (near-)real-time market data, and...
research
03/07/2020

Aion: Better Late than Never in Event-Time Streams

Processing data streams in near real-time is an increasingly important t...
research
01/04/2019

Intelligent Intersection: Two-Stream Convolutional Networks for Real-time Near Accident Detection in Traffic Video

In Intelligent Transportation System, real-time systems that monitor and...
research
05/03/2021

Stream Compression of DLMS Smart Meter Readings

Smart electricity meters typically upload readings a few times a day. Ut...
research
09/02/2019

Avaya Conversational Intelligence: A Real-Time System for Spoken Language Understanding in Human-Human Call Center Conversations

Avaya Conversational Intelligence(ACI) is an end-to-end, cloud-based sol...
research
07/28/2023

FleXR: A System Enabling Flexibly Distributed Extended Reality

Extended reality (XR) applications require computationally demanding fun...
research
05/09/2023

Instant-NeRF: Instant On-Device Neural Radiance Field Training via Algorithm-Accelerator Co-Designed Near-Memory Processing

Instant on-device Neural Radiance Fields (NeRFs) are in growing demand f...

Please sign up or login with your details

Forgot password? Click here to reset