The Event Stream Debacle
Unless you were hiding under a rock, you probably heard about the “event-stream situation” on NPM this week. TLDR: the original maintainer of the event-stream package was tired of maintaining it. He handed over the reins to a different developer, who promptly injected malware and released a new version.
Much of the ink spilled on this subject focused around open source expectations and who. was. responsible. for. this. happening. I don’t want to spend much time on this part of the discussion, so I’ll keep it short by saying the entitlement on display in that Github issue was remarkable. Imagine a scenario where someone gave $20 to everyone who wanted it. Some folks in that thread would be angry because it wasn’t $50.
What I do want to spend time talking about is why this happened. And why does it keep happening to Node/NPM?
“Vendoring” is for the Olds
Before Node and NPM, web applications didn’t usually have gobs of dependencies. You might have jQuery, a few plugins, and maybe a dozen or so other helper libraries. Every dependency in your app could cause bugs, break, have security problems, or be a pain to maintain and upgrade. You didn’t take a dependency unless you had to. Dependencies haven’t gotten magically safer in 2018. All these risks are still present, but some folks have forgotten the message.
There was also no package manager to install the dependency for you. If you wanted to use third party code, you pulled up the file in your browser, hit Save As, and put it in a lib folder somewhere. It was sort of a pain, but it worked. We didn’t have a term for it 10 years ago, but these days we call it vendoring.
And a lot of people really hate it. It’s cumbersome to include a dependency when you’re vendoring! You have to save it, include it in your build process, make sure it’s committed to source control etc. There’s no easy way to upgrade that dependency. There’s no easy way for that dependency to depend on other dependencies. It’s so archaic!
Thus, NPM was born.
One Package is Good, Four Hundred Must be Better
With NPM, JavaScript developers could make and take dependencies with a few keystrokes. Better still, they didn’t have to include all those pesky files in source control. Just npm install on a new box and you’re off to the races (ha ha).
Since it was easy to create and use dependencies, the JS developer community said, “no one should ever have to write a left padding function again!” And lo, we entered the era of micropackages. These are packages of only a few lines that do the barest minimum of functionality. It’s taking the Unix philosophy of “each tool doing one thing well”, and perverting it to the extreme. But too many packages is part of the reason things like left-pad and event-stream keep happening.
An Example From TrackJS
The TrackJS UI, until very recently, only used vendored JavaScript dependencies. We didn’t use Webpack, NPM or any “modern” JavaScript toolchain. We concated and minified our /js/lib folder, and it worked. Every. Single. Time.
But recently we needed to build a new piece of UI that was heavily client side interactive. We turned to React. We have a very basic “app”. We are only drawing a small part of our UI with it, and the whole thing is a handful of components. And still there are 418 dependencies in the node_modules folder! Just to draw a tiny bit of HTML. And look at some of these….
Ermm, is-arrayish? What’s the difference between is-plain-object and isobject? Every one of these is another avenue for bugs or security issues to creep in.
The ^ of Doom
So anyways, it’s now the norm for modern JavaScript apps to have hundreds of dependencies. And we’re not vendoring dependencies anymore - we just pull them in on-demand with the package manager. But there is yet a third problem with the culture of NPM and Node, one that exacerbates those other two: It is expected that you will upgrade your dependencies constantly (and that upgrading is always a desirable activity).
The default, when including a new package with NPM, is to specify it with the ^ operator (most recent minor version.)
"react": "^16.6.0",
This means whenever someone does a fresh install of this application they will get the most recent version of React that has a major version of 16. It could be React 16.6.0, 16.9.5 or 16.154.45. If the library in question is using semantic versioning, there should be no intentional breaking changes between minor versions. But, in practice, that can never be counted on. Additionally, bugs surface frequently between minor versions. And anyways, there’s no rule that says a package maintainer has to use semver. If I want to inject a bitcoin wallet snatcher in a point release, by golly I will.
The scariest part is that you will also be retrieving the latest version of that package’s required dependencies. Perhaps those have been updated too! And maybe in ways that will break your app. And those kind of bugs and breaks are extra painful to track down. The layers of indirection are enough to make you scream.
No One Knows You’re a Cryptominer on The Internet
The final act in the event-stream saga was that a reasonably well-known OSS contributor handed control of a package to a completely unknown developer (who then did bad things). Open source software has many excellent qualities, but vetting and verifying the backgrounds of contributors is not one of them (nor should it be!) The fact that critical software in the Node/JavaScript ecosystem is developed by anonymous individuals and mostly works is pretty amazing. But, whenever you’re relying on contributions from millions of individuals who aren’t getting paid, you’re going to get some bad apples. For every third party dependency you use, you’re putting your trust in that author. You are trusting them to run code on your servers. Hopefully their incentives align with yours!
The Grass is Greener… on Microsoft’s side?
Contrast the dependency situation in Node with that of .Net. Our TrackJS web app is built with ASP.NET MVC and is roughly 6 years old. It’s tens of thousands of lines of code. It does a tremendous number of things. And to accomplish all of it, we have just over 40 dependencies managed with Nuget (the NPM for .Net). Of those, 25 are written and maintained by Microsoft. Another dozen are written and maintained by the companies whose APIs they integrate with (Stripe, AWS etc). We have under 10 dependencies written by anonymous open source contributors in the entirety of our (large) .Net code base. And for those we do use, not one has child dependencies.
The difference here is staggering. A 6-year-old, massive by comparison, .Net app has 10 times fewer dependencies than a 1-month-old handful of basic React components. On top of that, 80% of the .Net dependencies are built by companies with incentives to keep quality high, and the bitcoin mining to a minimum.
A Brighter Future?
If you combine hundreds of dependencies, pulled in from random github repositories maintained by anonymous developers, with constant upgrades, you’re going to end up with problems like left-pad and event-stream. But things are going (slowly) in the right direction. As of NPM 5 there is a package.lock.json file created every time you install a new dependency. It’s basically the same thing as npm shrinkwrap except it happens automatically. If you check the file in, and another developer does an npm install on their box, in theory, they should get the exact same dependency tree as you, version for version.
JavaScript developers are starting to re-remember that dependencies aren’t free. If enough of these incidents occur perhaps the culture will start to value fewer packages, especially those built and maintained by trusted third parties? Maybe we don’t need an is-array and an is-arrayish? Maybe it’s better to copy and paste those one-liners off Stack Overflow? And maybe it’s better to use packages created and maintained by companies that have a profit motive to do so?
Until the developer culture changes, your app will continue to break for all manner of reasons. If you want instant notifications for when you app is having problems, give TrackJS a try!
