Speeding up the JavaScript ecosystem - Polyfills gone rogue

In the previous posts we looked at runtime performance and I thought it would be fun to look at node modules install time instead. A lot has been already written about various algorithmic optimizations or using more performant syscalls, but why do we even have this problem in the first place? Why is every node_modules folders so big? Where are all these dependencies coming from?

It all started when a buddy of mine noticed something odd with the dependency tree of his project. Whenever he updated a dependency, it would pull in several new ones and with each subsequent update the total number of dependencies grew over time.

 ├─┬ arraybuffer.prototype.slice 1.0.2
 │ └─┬ define-properties 1.2.1
 │   └── define-data-property 1.1.0
 ├─┬ function.prototype.name 1.1.6
 │ └─┬ define-properties 1.2.1
 │   └── define-data-property 1.1.0
 ├─┬ globalthis 1.0.3
 │ └─┬ define-properties 1.2.1
 │   └── define-data-property 1.1.0
 ├─┬ object.assign 4.1.4
 │ └─┬ define-properties 1.2.1
 │   └── define-data-property 1.1.0
 ├─┬ regexp.prototype.flags 1.5.1
 │ ├─┬ define-properties 1.2.1
 │ │ └── define-data-property 1.1.0
 │ └─┬ set-function-name 2.0.1
 │   └── define-data-property 1.1.0
 ├─┬ string.prototype.trim 1.2.8
 │ └─┬ define-properties 1.2.1
 │   └── define-data-property 1.1.0
 ├─┬ string.prototype.trimend 1.0.7
 │ └─┬ define-properties 1.2.1
 │   └── define-data-property 1.1.0
 └─┬ string.prototype.trimstart 1.0.7
   └─┬ define-properties 1.2.1
 	└── define-data-property 1.1.0

Now to be fair, there are valid reasons why a package might depend on an additional dependencies. Here, we began to notice a pattern though: The new dependencies were all polyfills for JavaScript functions that have long been supported everywhere. The Object.defineProperties method for example was shipped as part of the very first public Node 0.10.0 release dating back to 2013. Heck, even Internet Explorer 9 supported that. And yet there were numerous packages in that dependend on a polyfill for it.

Among the various packages that pulled in define-properties was eslint-plugin-react. It caught my eye, because it's very popular in the React ecosystem. Why does it pull in a polyfill for Object.defineProperties? There is no JavaScript engine that doesn't come with it already built in.

Polyfills that don’t polyfill

Reading the source of the packages revealed something even more bizarre: The polyfill functions were imported and called directly, rather than patching missing functionality in the runtime environment. The whole point of a polyfill is to be transparent to the user’s code. It should check if the function or method to patch is available and only add it if it’s missing. When there is no need for the polyfill it should do nothing. The odd thing to me is that the functions were used directly like a function from a library.

// Why is the `define` function imported directly?
var define = require("define-properties");
// ...

// and even worse, why is called directly?
define(polyfill, {
	getPolyfill: getPolyfill,
	implementation: implementation,
	shim: shim,
});

Instead they should call Object.defineProperties directly. The whole point of polyfills is to patch the environment not be called directly. Compare this to what a polyfill for Object.defineProperties is supposed to look like:

// Check if the current environment already supports
// `Object.defineProperties`. If it does, then we do nothing.
if (!Object.defineProperties) {
	// Patch in Object.defineProperties here
}

The most common place where define-properties was used was ironically inside other polyfills, which loaded even more polyfills. And before you ask, the define-properties package relies on even more dependencies than just itself.

var keys = require("object-keys");
// ...
var defineDataProperty = require("define-data-property");
var supportsDescriptors = require("has-property-descriptors")();

var defineProperties = function (object, map) {
	// ...
};

module.exports = defineProperties;

Inside eslint-plugin-react, that polyfill is loaded via a polyfill for Object.entries() to process their configuration:

const fromEntries = require("object.fromentries"); // <- Why is this used directly?
const entries = require("object.entries"); // <- Why is this used directly?
const allRules = require("../lib/rules");

function filterRules(rules, predicate) {
	return fromEntries(entries(rules).filter(entry => predicate(entry[1])));
}

const activeRules = filterRules(allRules, rule => !rule.meta.deprecated);

Doing a little bit of housekeeping

At the time of this writing installing eslint-plugin-react pulls in a whopping number of 97 dependencies in total. I was curious about how much of these were polyfills and began patching them out one by one locally. After all was done, this brought down the total number of dependencies down to 15. Out of the original 97 dependencies 82 of them are not needed.

Coincidentally, eslint-plugin-import which is equally popular in various eslint presets, shows a similar problems. Installing that fills up your node_modules folder with 87 packages. After another local cleanup pass I was able to cut down that number to just 17.

Filling up everyone's disk space

Now you might be wondering if you’re affected or not. I did a quick search and basically every widely popular eslint plugin or preset that you can think of is affected. For some reason this whole ordeal reminds me of the is-even/is-odd incident the industry had a while back.

Having so many dependencies makes it much harder to audit the dependencies of a project. It's a waste of space too. Case in point: Deleting all eslint plugins and presets in a project alone got rid of 220 packages.

pnpm -r rm eslint-plugin-react eslint-plugin-import eslint-import-resolver-typescript eslint-config-next eslint @typescript-eslint/parser @typescript-eslint/eslint-plugin eslint-plugin-prettier prettier eslint-config-prettier eslint-plugin-react-hooks
Scope: all 8 workspace projects
.                                        | -220 ----------------------

Maybe we don't need that many dependencies in the first place. My mind went to this fantastic quote by the creator of the Erlang programming language:

You wanted a banana but what you got was a gorilla holding the banana and the entire jungle - Joe Armstrong

All I wanted was some linting rules. I didn’t want a bunch of polyfills that I don't need.