Converting a Node Project from CommonJS to ESM

I had a small JavaScript project (deployed via AWS lambda, which I’ll write about later on). I needed to add a very simple package to the project—detect-cloudflare—which sent me down a deep rabbit hole:

• My project was written years ago using CommonJS (i.e. require).
• cloudflare-detect included a bunch of packages for a very simple task, which bothers me. Plus, the IP address ranges were out of date and the package hadn’t been updated in years, I decided I wanted to update it. Should be easy, right? (this is never a good thing for an engineer to say)
• I updated the project and got all of the tests passing without too much effort. Nice!
• When I went to use the newly refactored package in my project it failed with this error.

Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/mike/Projects/javascript/cloudflare-detect/node_modules/humanize-url/index.js from /Users/mike/Projects/javascript/cloudflare-detect/dist/index.js not supported.

Uh oh. This is where my problems began.

Like all things Javascript, this was way more time consuming and painful that I could have imagined.

Converting an ESM package to CommonJS

This seems like the most straightforward path.

• Babel can convert an ESM project to CommonJS format. Set "@babel/preset-env" with "modules": "commonjs and then run babel npx babel ./src/ --out-dir dist.
  • Worth noting that the correct syntax for this transpilation has changed a lot over the years. It took me a while to realize that no additional babel packages were needed outside of @babel/preset-env (at least of this writing!)
• However, if your package uses an ESM-only package, babel will not convert the underlying package to be CJS compatible and no errors are emitted.
  • It was unclear to me how to run your test suite against both the ESM and CJS versions of a package. I wonder what the best practices are here since ESM & CJS packages are not identical (they could fail because of differences in how the underlying packages generate the ESM & CJS versions of their packages).
• In my case, detect-cloudflare had a dependency (which was annoyingly simple) but required another package that switched to ESM only. You can imagine how fixing ESM-only issues in this dependency graph is a huge lift.

Even though I had the CommonJS and ESM versions of the package properly emitted during build (this is a great guide) I decided it was a better idea to convert the project to use ESM to avoid this issue in the future.

What does it take to convert a CommonJS project to ESM

ESM is definitely a better module system, but converting a project to ESM is non-trivial. This is a great guide to converting a CJS project to ESM.

Here are some problems you run into:

• Some packages support CJS and not ESM. Even really popular packages like lodash.
  • For popular packages there are often ESM generated versions of the package. Here‘s the one for lodash.
  • This creates additional complexity when using a tool to help rewrite imports: instead of * as _ you’ll want to use just use _ since lodash-es has a default export (where the standard package does not)
• There’s a neat package cjstoesm. However, don’t get too excited:
  • Comments are sometimes mutated
  • Empty lines (newline + whitespace) are removed from the source
  • It doesn’t work with the latest typescript
• cjstoesm doesn’t handle everything
  • use strict is not removed from the source
  • Anon functions using exports.functionName = () => {} failed to convert for me

However, even with these caveats it felt worth it to do the conversion. My project was relatively small otherwise this would be a massive pain.

Converting CommonJS to ESM

Here’s the process I used to convert the project.

• First, setup prettier or another code formatter and format all your JS. Then commit any changes to your repo.
• Run the automatic conversion: fd . --extension=js --type=file --threads=1 --exec zsh -c "npx -p typescript@4.9.5 -p cjstoesm cjstoesm {}"
  • Note that fd is used here to make sure js in node_modules isn’t picked up
• Run git add -p and manually commit the import + export changes in the file, ignoring all of the newline removals.
• Switch your package.json to "type": "module",
• Note that some files were not converted using the fd-based script above. Some exports.functionName = () => {} style functions did not convert either. Handle these manually.
• Remove use strict from all of the files fd . --extension=js --type=file --exec sed -i '' '/; *("use strict")/d' {}
• Try running your tests and other code and this will identify any packages that don’t support ESM. This is where all of the danger is. Good luck.

Installing NPM Packages From GitHub

It seems (at least on npm version 9.8.1) if you are installing a npm repo from github using:

"cloudflare-detect": "github:iloveitaly/cloudflare-detect#large-refactor",

npm can get ‘stuck’ on a revision on that branch. In other words, if you are iterating on the package and updating the branch frequently it can reference an older package.json.

Specifying a SHA reference, running npm install, and then reverting back to the branch reference fixed the issue for me. This may have been caused by using npm link to iterate on the package locally.

In the future, my plan will be:

1. Use file: reference in the package.json in combination with npm link when iterating
2. When finished, wipe node_modules and switch to a branch git + branch reference in package.json