DeepAI AI Chat
Log In Sign Up

So Much in So Little: Creating Lightweight Embeddings of Python Libraries

by   Yaroslav Golubev, et al.
HUAWEI Technologies Co., Ltd.

In software engineering, different approaches and machine learning models leverage different types of data: source code, textual information, historical data. An important part of any project is its dependencies. The list of dependencies is relatively small but carries a lot of semantics with it, which can be used to compare projects or make judgements about them. In this paper, we focus on Python projects and their PyPi dependencies in the form of requirements.txt files. We compile a dataset of 7,132 Python projects and their dependencies, as well as use Git to pull their versions from previous years. Using this data, we build 32-dimensional embeddings of libraries by applying Singular Value Decomposition to the co-occurrence matrix of projects and libraries. We then cluster the embeddings and study their semantic relations. To showcase the usefulness of such lightweight library embeddings, we introduce a prototype tool for suggesting relevant libraries to a given project. The tool computes project embeddings and uses dependencies of projects with similar embeddings to form suggestions. To compare different library recommenders, we have created a benchmark based on the evolution of dependency sets in open-source projects. Approaches based on the created embeddings significantly outperform the baseline of showing the most popular libraries in a given year. We have also conducted a user study that showed that the suggestions differ in quality for different project domains and that even relevant suggestions might be not particularly useful. Finally, to facilitate potentially more useful recommendations, we extended the recommender system with an option to suggest rarer libraries.


page 1

page 7


DUETS: A Dataset of Reproducible Pairs ofJava Library-Clients

Software engineering researchers look for software artifacts to study th...

Import2vec - Learning Embeddings for Software Libraries

We consider the problem of developing suitable learning representations ...

Mining Threat Intelligence about Open-Source Projects and Libraries from Code Repository Issues and Bug Reports

Open-Source Projects and Libraries are being used in software developmen...

An Empirical Study of Library Usage and Dependency in Deep Learning Frameworks

Recent advances in deep learning (dl) have led to the release of several...

A Longitudinal Analysis of Bloated Java Dependencies

We study the evolution and impact of bloated dependencies in a single so...

Req2Lib: A Semantic Neural Model for Software Library Recommendation

Third-party libraries are crucial to the development of software project...

Hero: On the Chaos When PATH Meets Modules

Ever since its first release in 2009, the Go programming language (Golan...