Python Implementation of the Dynamic Distributed Dimensional Data Model

09/01/2022
by   Hayden Jananthan, et al.
0

Python has become a standard scientific computing language with fast-growing support of machine learning and data analysis modules, as well as an increasing usage of big data. The Dynamic Distributed Dimensional Data Model (D4M) offers a highly composable, unified data model with strong performance built to handle big data fast and efficiently. In this work we present an implementation of D4M in Python. D4M .py implements all foundational functionality of D4M and includes Accumulo and SQL database support via Graphulo. We describe the mathematical background and motivation, an explanation of the approaches made for its fundamental functions and building blocks, and performance results which compare D4M .py's performance to D4M-MATLAB and D4M.jl

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/14/2016

Julia Implementation of the Dynamic Distributed Dimensional Data Model

Julia is a new language for writing data analysis programs that are easy...
research
05/22/2019

AXS: A framework for fast astronomical data processing based on Apache Spark

We introduce AXS (Astronomy eXtensions for Spark), a scalable open-sourc...
research
09/24/2016

Benchmarking SciDB Data Import on HPC Systems

SciDB is a scalable, computational database management system that uses ...
research
02/18/2021

A Unified System for Data Analytics and In Situ Query Processing

In today's world data is being generated at a high rate due to which it ...
research
11/21/2017

HybridTune: Spatio-temporal Data and Model Driven Performance Diagnosis for Big Data Systems

With tremendous growing interests in Big Data systems, analyzing and fac...
research
04/07/2021

Efficient and Accurate In-Database Machine Learning with SQL Code Generation in Python

Following an analysis of the advantages of SQL-based Machine Learning (M...
research
07/06/2019

Streaming 1.9 Billion Hypersparse Network Updates per Second with D4M

The Dynamic Distributed Dimensional Data Model (D4M) library implements ...

Please sign up or login with your details

Forgot password? Click here to reset