Bootstrapping ClojureScript compiler for live coding environment

The core of the studio's experience is the ClojureScript compiler. It is where your code is being compiled before running in a browser. ClojureScript compiler got bootstrapping capability some years ago and was used in many playgrounds since then, ClojureScript Studio is no exception. However bootstrapping the compiler is not the only thing that is required to provide a good live coding experience.

Bootstrapping a compiler means compiling it using the compiler itself into a program that runs in target environment, which in this case is JavaScript. Luckily it's an easy task these days with shadow-cljs.

{:builds
 {:runtime {:target :browser
            :output-dir "resources/public/out"
            :asset-path "/out"
            :modules {:runtime {:entries [studio.runtime]}}
            :js-options {:minimize-require false}
            :compiler-options {:output-wrapper false}
            :release {:compiler-options {:optimizations :simple}}}

  :bootstrap {:target :bootstrap
              :output-dir "resources/public/out/bootstrap"
              :exclude #{cljs.js}
              :entries [cljs.js
                        reagent.core reagent.dom
                        re-frame.core
                        "react"
                        "react-dom"
                        "react-dom/client"]
              :js-options {:minimize-require false}
              :compiler-options {:output-wrapper false}}}}

For in-depth overview of the build config I'd recommend to read the article from shadow's author on this topic.

Besides cljs compiler, the :bootstrap config also includes some dependencies from Clojars and NPM. Those are the libraries that I want to have available in the runtime environment, so that you can create UIs with Reagent and re-frame for example. Technically bundling those libraries into the compiler is not required, but it's an easy way to make sure that they are available. On the other hand it means I have to hand pick dependencies, which is not ideal.

Let's focus on NPM deps here. They are not relly a part of ClojureScript and are not dependant on anything in the compiler. But to make it possible to require NPM deps from ClojureScript, the compiler should be able to make sense of JavaScript's modules. That's what shadow is doing and that's why it's just easier to bundle those deps with the compiler. During compilation step all JS modules are wrapped with a gluing code that exposes modules as namespaces, thus making them consumable from ClojureScript.

// the wrapper
shadow$provide["@react-three/fiber"] = function (
  global,
  require,
  module,
  exports
) {
  // JS module from NPM here that export a bunch of stuff
  module.exports = {};
};
// declare Closure namespace
goog.provide("@react-three/fiber");
// pull the module into Closure namespace from shadow's index
goog.global["@react-three/fiber"] = shadow.js.require("@react-three/fiber", {});

This can be done outside of compilation step, even at runtime. Currently I'm exploring an option of pulling in any library from NPM into the studio. I'm using UNPKG as a CDN for NPM packages and Cloudflare Workers to run libraries through Babel on demand, just in case there's some JS that is not widely supported yet or a module is in ESM format, and wrap modules with the above code.

In fact, when compiled with shadow, generated namespace would look something like this module$node_modules$react$index, but I've chosen to just use package name from NPM as a namespace. This way I don't have to rewrite requires in user code at runtime.

This approach of rewriting JS modules requires additional work to resolve dependency tree, but it doesn't mean it's not possible to build such a system.