The npm-accel program is a wrapper for npm (the Node.js package manager) that was created to optimize one specific use case: Creating a node_modules directory from a package.json file as quickly as possible.
It works on the assumption that you build node_modules directories more frequently then you change the contents of package.json files, because it computes a fingerprint of the dependencies and uses that fingerprint as a cache key, to cache the complete node_modules directory in a tar archive.
The program is intended to be used in environments that always or frequently start with an empty node_modules directory and need to populate the complete directory from scratch (e.g. continuous integration builds and deployments). I'm specifically not claiming that you will see any speed improvements if you're updating existing node_modules directories.
The npm-accel program is currently tested on Python 2.6, 2.7, 3.4, 3.5 and PyPy (yes, it's written in Python, deal with it :-P). It's intended to work on UNIX systems like Linux and Mac OS X and specifically won't work on Windows (see supported operating systems for details).
The npm-accel project was developed and published in September '16 because I got fed up waiting for npm install
to finish, specifically in the context of continuous integration builds and deployments (where you frequently start with an empty node_modules
directory). It was developed in about a week without much prior knowledge about Node.js or npm, which explains why it's written in Python :-P. On the one hand npm-accel hasn't seen much actual use, on the other hand it has a test suite with about 95% test coverage and I was careful not to repeat the bugs I encountered in npm-cache and npm-fast-install while evaluating those tools :-).
To summarize: Give it a try, see if it actually speeds up your npm install
use case and then decide whether you want to use it or not. The first releases of npm-accel are labeled as alpha releases because the program hasn't seen much real world use (and I'm no expert in Node.js and npm).
The following table lists the output of npm-accel --benchmark
(with some enhancements) against a private code base with about 30 dependencies listed in the package.json file (resulting in a 401 MB node_modules directory):
Approach | Iteration | Elapsed time | Percentage |
---|---|---|---|
npm install |
|
1 minute and 36 seconds |
|
npm install |
|
1 minute and 40 seconds |
|
npm-accel |
|
1 minute and 40 seconds |
|
npm-accel |
|
|
|
npm-cache install npm |
|
3 minutes and 9 seconds |
|
npm-cache install npm |
|
2 minutes and 9 seconds |
|
npm-fast-install |
|
1 minute and 5 seconds |
|
npm-fast-install |
|
1 minute and 6 seconds |
|
Some notes about this benchmark:
- Each of the four installation methods is run twice. The first run starts with empty cache directories and is intended to "prime the cache". The second run is intended to actually use the cache and should be able to do so quite effectively, given that the package.json file does not change between the two runs.
- During the benchmark, the caching performed by npm-accel is only used in the fourth row of the table above. This is because the original point of the benchmark (for me) was to find out whether it was even worth it to develop and publish npm-accel. That is to say, if it wouldn't have given a speed improvement it wasn't worth my time, nor yours :-P.
The npm-accel package is available on PyPI which means installation should be as simple as:
$ pip install npm-accel
There's actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).
Alternatively if you are running Arch Linux, npm-accel is available in AUR.
There are two ways to use the npm-accel package: As the command line program npm-accel
and as a Python API. For details about the Python API please refer to the API documentation available on Read the Docs. The command line interface is described below.
Usage: npm-accel [OPTIONS] [DIRECTORY]
The npm-accel program is a wrapper for npm (the Node.js package manager) that optimizes one specific use case: Building a "node_modules" directory from a "package.json" file as quickly as possible.
It works on the assumption that you build "node_modules" directories more frequently then you change the contents of "package.json" files, because it computes a fingerprint of the dependencies and uses that fingerprint as a cache key, to cache the complete "node_modules" directory in a tar archive.
Supported options:
- Accelerate installations with changes
Currently when the fingerprint (cache key) of the dependencies doesn't match a cache entry, the complete caching mechanism is bypassed and a full
npm install
run is performed. It might be faster to unpack a previous (now invalid) cache entry corresponding to the same project and then runnpm install && npm prune
. Given the fact that defining "same project" might be non-trivial I'm not actually sure this is worth my time.- Dealing with optionalDependencies
I've never seen
optionalDependencies
in the wild but encountered them while browsing through the package.json documentation. Maybe these should be part of the computed cache keys aswell?
The npm-accel program was developed to work on UNIX systems like Linux and Mac OS X. It requires several external commands to be installed (e.g. mkdir
, mv
, rm
, tar
and which
).
I've tried to keep all of the external command invocations compatible with the Linux and BSD variants of commands like tar, that is to say npm-accel uses only short options and skips the more esoteric features even if they would be useful. If you find that I failed in this respect, please feel free to report this as a bug.
For posterity: It was a conscious decision (for several reasons) to use the tar program instead of manipulating tar archives via Python's tarfile module.
The latest version of npm-accel is available on PyPI and GitHub. The documentation is hosted on Read the Docs. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.
This software is licensed under the MIT license.
© 2016 Peter Odding.