DeepAI AI Chat
Log In Sign Up

The Ghost of Performance Reproducibility Past

by   Srinivasan Ramesh, et al.
University of Oregon
Brookhaven National Laboratory

The importance of ensemble computing is well established. However, executing ensembles at scale introduces interesting performance fluctuations that have not been well investigated. In this paper, we trace our experience uncovering performance fluctuations of ensemble applications (primarily constituting a workflow of GROMACS tasks), and unsuccessful attempts, so far, at trying to discern the underlying cause(s) of performance fluctuations. Is the failure to discern the causative or contributing factors a failure of capability? Or imagination? Do the fluctuations have their genesis in some inscrutable aspect of the system or software? Does it warrant a fundamental reassessment and rethinking of how we assume and conceptualize performance reproducibility? Answers to these questions are not straightforward, nor are they immediate or obvious. We conclude with a discussion about the performance of ensemble applications and ruminate over the implications for how we define and measure application performance.


page 1

page 2

page 3

page 4


Reproducibility in Optimization: Theoretical Framework and Limits

We initiate a formal study of reproducibility in optimization. We define...

Diversity and Generalization in Neural Network Ensembles

Ensembles are widely used in machine learning and, usually, provide stat...

Use Cases of Computational Reproducibility for Scientific Workflows at Exascale

We propose an approach for improved reproducibility that includes captur...

Reproducibility and Performance: Why Choose?

Research processes often rely on high-performance computing (HPC), but H...

SwarmRob: A Toolkit for Reproducibility and Sharing of Experimental Artifacts in Robotics Research

Due to the complexity of robotics, the reproducibility of results and ex...

Forming Ensembles at Runtime: A Machine Learning Approach

Smart system applications (SSAs) built on top of cyber-physical and soci...

Distributed Ensembles of Reinforcement Learning Agents for Electricity Control

Deep Reinforcement Learning (or just "RL") is gaining popularity for ind...