Toward Efficient Interactions between Python and Native Libraries

by   Jialiang Tan, et al.

Python has become a popular programming language because of its excellent programmability. Many modern software packages utilize Python for high-level algorithm design and depend on native libraries written in C/C++/Fortran for efficient computation kernels. Interaction between Python code and native libraries introduces performance losses because of the abstraction lying on the boundary of Python and native libraries. On the one side, Python code, typically run with interpretation, is disjoint from its execution behavior. On the other side, native libraries do not include program semantics to understand algorithm defects. To understand the interaction inefficiencies, we extensively study a large collection of Python software packages and categorize them according to the root causes of inefficiencies. We extract two inefficiency patterns that are common in interaction inefficiencies. Based on these patterns, we develop PieProf, a lightweight profiler, to pinpoint interaction inefficiencies in Python applications. The principle of PieProf is to measure the inefficiencies in the native execution and associate inefficiencies with high-level Python code to provide a holistic view. Guided by PieProf, we optimize 17 real-world applications, yielding speedups up to 6.3× on application level.




Toward Interlanguage Parallel Scripting for Distributed-Memory Scientific Computing

Scripting languages such as Python and R have been widely adopted as too...

Using Python for Model Inference in Deep Learning

Python has become the de-facto language for training deep neural network...

PyDraw: a GUI drawing generator based on Tkinter and its design concept

The emergence of GUI is a great progress in the history of computer scie...

PyXNAT: XNAT in Python

As neuroimaging databases grow in size and complexity, the time research...

Scalene: Scripting-Language Aware Profiling for Python

Existing profilers for scripting languages (a.k.a. "glue" languages) lik...

Gobi: WebAssembly as a Practical Path to Library Sandboxing

Software based fault isolation (SFI) is a powerful approach to reduce th...

Pytrec_eval: An Extremely Fast Python Interface to trec_eval

We introduce pytrec_eval, a Python interface to the tree_eval informatio...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.