DeepAI AI Chat
Log In Sign Up

Weld: Rethinking the Interface Between Data-Intensive Applications

by   Shoumik Palkar, et al.

Data analytics applications combine multiple functions from different libraries and frameworks. Even when each function is optimized in isolation, the performance of the combined application can be an order of magnitude below hardware limits due to extensive data movement across these functions. To address this problem, we propose Weld, a new interface between data-intensive libraries that can optimize across disjoint libraries and functions. Weld exposes a lazily-evaluated API where diverse functions can submit their computations in a simple but general intermediate representation that captures their data-parallel structure. It then optimizes data movement across these functions and emits efficient code for diverse hardware. Weld can be integrated into existing frameworks such as Spark, TensorFlow, Pandas and NumPy without changing their user-facing APIs. We demonstrate that Weld can speed up applications using these frameworks by up to 29x.


page 1

page 2

page 3

page 4


Splitability Annotations: Optimizing Black-Box Function Composition in Existing Libraries

Data movement is a major bottleneck in parallel data-intensive applicati...

UCX Programming Interface for Remote Function Injection and Invocation

Network library APIs have historically been developed with the emphasis ...

An Empirical Study of Library Usage and Dependency in Deep Learning Frameworks

Recent advances in deep learning (dl) have led to the release of several...

BrainSlug: Transparent Acceleration of Deep Learning Through Depth-First Parallelism

Neural network frameworks such as PyTorch and TensorFlow are the workhor...

A Data-Centric Optimization Framework for Machine Learning

Rapid progress in deep learning is leading to a diverse set of quickly c...

SOL: Reducing the Maintenance Overhead for Integrating Hardware Support into AI Frameworks

The increased interest in Artificial Intelligence (AI) raised the need f...