Distributed-Something: scripts to leverage AWS storage and computing for distributed workflows at scale

10/03/2022
by   Erin Weisbart, et al.
0

Distributed-Something coordinates the distribution of any Dockerized workflow using on-demand computational infrastructure from Amazon Web Services to enable at-scale workflows where neither computing power nor data storage are limited by local availability while minimizing the time-consuming and confusing aspects of architecture coordination. We also provide Distributed-Something implementations of several bioimaging tools: Distributed-CellProfiler, -Fiji, and -OmeZarrCreator. All are open-source and available at http://GitHub.com/DistributedScience.

READ FULL TEXT
research
08/05/2019

Data Aggregation In The Astroparticle Physics Distributed Data Storage

German-Russian Astroparticle Data Life Cycle Initiative is an internatio...
research
05/16/2019

StashCache: A Distributed Caching Federation for the Open Science Grid

Data distribution for opportunistic users is challenging as they neither...
research
07/04/2022

ImageBox3: No-Server Tile Serving to Traverse Whole Slide Images on the Web

Whole slide imaging (WSI) has become the primary modality for digital pa...
research
04/28/2023

Servo: Increasing the Scalability of Modifiable Virtual Environments Using Serverless Computing – Extended Technical Report

Online games with modifiable virtual environments (MVEs) have become hig...
research
05/28/2015

Visual Search at Pinterest

We demonstrate that, with the availability of distributed computation pl...
research
09/16/2021

BuDDI: A Declarative Bloom Language for CALM Programming

Coordination protocols help programmers of distributed systems reason ab...
research
03/13/2023

Integration of storage endpoints into a Rucio data lake, as an activity to prototype a SKA Regional Centres Network

The Square Kilometre Array (SKA) infrastructure will consist of two radi...

Please sign up or login with your details

Forgot password? Click here to reset