tl;dr Node.js and browsers are different Javascript-based platforms offering different APIs. How do you write Scala libraries that can be used on either platform? Could something like crossProject(JVMPlatform, NodeJSPlatform, WebAPIJSPlatform) be a way forward?
Prologue
I’ve spent the last few weeks cross-building http4s for Scala.js, along with several of its upstream dependencies, most notably fs2.io. Although ultimately any single implementation of an http4s client/server will target Node.js or the browser, to enable this, the core and several middleware modules must be able to run in either environment. This has been an interesting and perhaps ambitious experiment: one of the most interesting outcomes (in my opinion) has been the (very nearly) pure cross of the http4s ember server/client to SJS enabled by implementing the fs2․io Socket[F] trait on Node.js.
I want to share my experience and ask for feedback on the one of the biggest impediments I encountered working on this project. I should note, this is my first substantial experience with SJS and so I am by no means an expert nor aware of all the great projects and techniques in this ecosystem.
The problem
As stated in the tl;dr: Node.js and browsers are different Javascript-based platforms offering different APIs (“standard libraries”). How do you write Scala libraries that run on both of these platforms, while also taking advantage of these platform-specific APIs?
Workaround
The go-to workaround is the dynamic use of require (comparable I think to attempting to load a class via reflection). This is a useful hack, but it quickly breaks down when you want to start including dependencies.
A case study: cryptography is essential to several http4s features, and crypto APIs are available on both Node.js and browsers. Fortunately, scala-js-dom provides facades for the browser, and we can auto-generate a Node.js facade with Scalably Typed.1
Alas, depending on the Scalably Typed Node.js facade would competely block the browser usecase due to the lack of support for the @JSImport annotation. But then, our luck holds: it seems the Web Cryptography API from browsers is also being offered experimentally in Node.js v16! So perhaps we can use the dynamic require trick after all, with the facade provided by scala-js-dom. This also turns out to be a dead-end, due to a rogue package object initializer in scala-js-dom that throws an error on Node.js. So, the last resort: copy-paste the scala-js-dom crypto facade into http4s and use a Try with dynamic require to find the available implementation at runtime.
As far as I can tell, the moral of this story is that for the dynamic require trick to work, all SJS libraries must be written in a way to accomodate this even if they are targeting only a single platform (Node.js or browser), because their downstream dependents may be targeting both of these platforms. This boxes most libraries into either choosing a single platform to support or restricting themselves to a feature set that they can support in both environments without using any platform-specific APIs.
Cross-builds as the principled solution?
The problem of maintaining a library across multiple APIs (aka standard libraries) and other incompatibilites while managing its dependencies has already been solved for Scala via cross-builds. Thus, I feel tempted to reach for this solution once again here, although I am also eager to hear how other projects have navigated this issue.
Specifically, I am curious if cross-builds for the Node.js–browser axis have ever been considered for SJS. Something like crossProject(JVMPlatform, NodeJSPlatform, WebAPIJSPlatform).2 Adding new platforms to a cross project seems fairly straightforward to me, but the bigger problem of course is solving this problem (if it exists?) at the community level so that we can keep using our magic %%% without being sent to dependency hell.
Conclusion
Strategic abstractions and cross-builds empower us to write even HTTP servers/clients in Scala agnostic to whether the sockets are provided by the JVM or Node.js. Yet, trying to navigate the smaller gap between the cryptography APIs in Node.js and the browser is painful in comparison. I would appreciate hearing from anyone else experiencing this pain in their own libraries or for whom a browser/Node.js crossing http4s would be a game-changer. Thanks for reading.
Footnotes
- An aside, but it is unfortunate that there is no standard Node.js facade for SJS, at least that I am aware of. For now, a Scalably Typed facade serves well in fs2․io.
- By no means to pile on, after making PRs to multiple projects upstream to http4s lacking SJS cross-builds, I am acutely aware of library authors’ frustration with the lack of first-class support for cross projects in SBT.
