npm-follower: A Complete Dataset Tracking the NPM Ecosystem

by   Donald Pinckney, et al.

Software developers typically rely upon a large network of dependencies to build their applications. For instance, the NPM package repository contains over 3 million packages and serves tens of billions of downloads weekly. Understanding the structure and nature of packages, dependencies, and published code requires datasets that provide researchers with easy access to metadata and code of packages. However, prior work on NPM dataset construction typically has two limitations: 1) only metadata is scraped, and 2) packages or versions that are deleted from NPM can not be scraped. Over 330,000 versions of packages were deleted from NPM between July 2022 and May 2023. This data is critical for researchers as it often pertains to important questions of security and malware. We present npm-follower, a dataset and crawling architecture which archives metadata and code of all packages and versions as they are published, and is thus able to retain data which is later deleted. The dataset currently includes over 35 million versions of packages, and grows at a rate of about 1 million versions per month. The dataset is designed to be easily used by researchers answering questions involving either metadata or program analysis. Both the code and dataset are available at


page 1

page 2

page 3

page 4


A Large Scale Analysis of Semantic Versioning in NPM

The NPM package repository contains over two million packages and serves...

Restoring Execution Environments of Jupyter Notebooks

More than ninety percent of published Jupyter notebooks do not state dep...

Practical Automated Detection of Malicious npm Packages

The npm registry is one of the pillars of the JavaScript and TypeScript ...

How Universal is Genre in Universal Dependencies?

This work provides the first in-depth analysis of genre in Universal Dep...

Learning Software Constraints via Installation Attempts

Modern software systems are expected to be secure and contain all the la...

Extending ROOT through Modules

The ROOT software framework is foundational for the HEP ecosystem, provi...

Analyzing the Evolution of Inter-package Dependencies in Operating Systems: A Case Study of Ubuntu

An Operating System (OS) combines multiple interdependent software packa...

Please sign up or login with your details

Forgot password? Click here to reset