An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

03/05/2023
by   Wenxin Jiang, et al.
0

Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as state-of-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems. In this work, we present the first empirical investigation of PTM reuse. We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. From this data, we model the decision-making process for PTM reuse. Based on the identified practices, we describe useful attributes for model reuse, including provenance, reproducibility, and portability. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks. We substantiate these identified challenges with systematic measurements in the Hugging Face ecosystem. Our work informs future directions on optimizing deep learning ecosystems by automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries.

READ FULL TEXT

page 1

page 9

research
03/15/2023

PTMTorrent: A Dataset for Mining Open-source Pre-trained Model Packages

Due to the cost of developing and training deep learning models from scr...
research
03/13/2023

Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision

Many engineering organizations are reimplementing and extending deep neu...
research
03/05/2023

Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability

Training deep neural networks (DNNs) takes signifcant time and resources...
research
08/17/2023

ZhiJian: A Unifying and Rapidly Deployable Toolbox for Pre-trained Model Reuse

The rapid expansion of foundation pre-trained models and their fine-tune...
research
12/02/2020

Empirical Study on the Software Engineering Practices in Open Source ML Package Repositories

Recent advances in Artificial Intelligence (AI), especially in Machine L...
research
02/04/2018

Software Engineers vs. Machine Learning Algorithms: An Empirical Study Assessing Performance and Reuse Tasks

Several papers have recently contained reports on applying machine learn...
research
11/30/2022

An Empirical Study on the Bugs Found while Reusing Pre-trained Natural Language Processing Models

In NLP, reusing pre-trained models instead of training from scratch has ...

Please sign up or login with your details

Forgot password? Click here to reset