Scala.js: how to support both Node.js and Browser platforms?

tl;dr Node.js and browsers are different Javascript-based platforms offering different APIs. How do you write Scala libraries that can be used on either platform? Could something like crossProject(JVMPlatform, NodeJSPlatform, WebAPIJSPlatform) be a way forward?

Prologue

I’ve spent the last few weeks cross-building http4s for Scala.js, along with several of its upstream dependencies, most notably fs2.io. Although ultimately any single implementation of an http4s client/server will target Node.js or the browser, to enable this, the core and several middleware modules must be able to run in either environment. This has been an interesting and perhaps ambitious experiment: one of the most interesting outcomes (in my opinion) has been the (very nearly) pure cross of the http4s ember server/client to SJS enabled by implementing the fs2․io Socket[F] trait on Node.js.

I want to share my experience and ask for feedback on the one of the biggest impediments I encountered working on this project. I should note, this is my first substantial experience with SJS and so I am by no means an expert nor aware of all the great projects and techniques in this ecosystem.

The problem

As stated in the tl;dr: Node.js and browsers are different Javascript-based platforms offering different APIs (“standard libraries”). How do you write Scala libraries that run on both of these platforms, while also taking advantage of these platform-specific APIs?

Workaround

The go-to workaround is the dynamic use of require (comparable I think to attempting to load a class via reflection). This is a useful hack, but it quickly breaks down when you want to start including dependencies.

A case study: cryptography is essential to several http4s features, and crypto APIs are available on both Node.js and browsers. Fortunately, scala-js-dom provides facades for the browser, and we can auto-generate a Node.js facade with Scalably Typed.1

Alas, depending on the Scalably Typed Node.js facade would competely block the browser usecase due to the lack of support for the @JSImport annotation. But then, our luck holds: it seems the Web Cryptography API from browsers is also being offered experimentally in Node.js v16! So perhaps we can use the dynamic require trick after all, with the facade provided by scala-js-dom. This also turns out to be a dead-end, due to a rogue package object initializer in scala-js-dom that throws an error on Node.js. So, the last resort: copy-paste the scala-js-dom crypto facade into http4s and use a Try with dynamic require to find the available implementation at runtime.

As far as I can tell, the moral of this story is that for the dynamic require trick to work, all SJS libraries must be written in a way to accomodate this even if they are targeting only a single platform (Node.js or browser), because their downstream dependents may be targeting both of these platforms. This boxes most libraries into either choosing a single platform to support or restricting themselves to a feature set that they can support in both environments without using any platform-specific APIs.

Cross-builds as the principled solution?

The problem of maintaining a library across multiple APIs (aka standard libraries) and other incompatibilites while managing its dependencies has already been solved for Scala via cross-builds. Thus, I feel tempted to reach for this solution once again here, although I am also eager to hear how other projects have navigated this issue.

Specifically, I am curious if cross-builds for the Node.js–browser axis have ever been considered for SJS. Something like crossProject(JVMPlatform, NodeJSPlatform, WebAPIJSPlatform).2 Adding new platforms to a cross project seems fairly straightforward to me, but the bigger problem of course is solving this problem (if it exists?) at the community level so that we can keep using our magic %%% without being sent to dependency hell.

Conclusion

Strategic abstractions and cross-builds empower us to write even HTTP servers/clients in Scala agnostic to whether the sockets are provided by the JVM or Node.js. Yet, trying to navigate the smaller gap between the cryptography APIs in Node.js and the browser is painful in comparison. I would appreciate hearing from anyone else experiencing this pain in their own libraries or for whom a browser/Node.js crossing http4s would be a game-changer. Thanks for reading.

Footnotes

  1. An aside, but it is unfortunate that there is no standard Node.js facade for SJS, at least that I am aware of. For now, a Scalably Typed facade serves well in fs2․io.
  2. By no means to pile on, after making PRs to multiple projects upstream to http4s lacking SJS cross-builds, I am acutely aware of library authors’ frustration with the lack of first-class support for cross projects in SBT.
5 Likes

I would add Browser extension - Wikipedia APIs to the list of targeted execution environments. However, I think a better place to put this elaborate question (to get Scala.js authors’ attention) is here Issues · scala-js/scala-js · GitHub or here Issues · portable-scala/sbt-crossproject · GitHub , because this thread has no traffic and it will quickly get out of sight here.

2 Likes

If this doesn’t get traction here it might make sense to open an issue for scala-js. It’s a big deal and it needs an actual solution.

1 Like

Oh, nice to see a couple responses here! Thank you both. I will definitely cross-post this somewhere else.

@tarsa AFAICT browser extension API is still not so stable at the moment. But the Web APIs and Node.js APIs are much more stable and standardized in comparison.

@tpolecat Sadly, I’m not sure how big of a deal this is for most projects. The projects I was crossing are in the very unusual position that (1) they want to support both browsers and Node.js and (2) platform-specific APIs (DOM, I/O, cryptography, etc.) are critical to their core features. Many projects fall into the first category (e.g., Cats, Circe) or the second (e.g., any UI framework, or say Skunk). But to need to do both at the same time, as http4s requires, is perhaps quite unusual.

I’m not sure I have enough practical experience with complicated situations like this to contribute meaningful advice. My gut reaction would be: try to stay away from cross-compiling at virtually all cost, as that explodes very quickly, so the ecosystem will get out of hands. Cross-compilation is viable when the entire ecosystem is uniformly cross-compiled. It’s not viable if some libraries are cross-compiled for browser/Node.js while others aren’t.

The problem of the “rogue val” in scalajs-dom is currently being resolved, with a def for scalajs-dom 1.x and with an @js.native val for scalajs-dom 2.x.

1 Like

Thank you for chiming in @sjrd, much appreciated :slight_smile:

The problem of the “rogue val” in scalajs-dom is currently being resolved,

That’s great, and for sure some of these problems can be solved on a case-by-case basis. For another example of this see munit #247. But overall this makes for a very frustrating experience. For example, if there is a library targeting Node.js currently using @JSImport style-facades and I want to use it to implement a feature for http4s core when running in Node.js, I cannot. The “case-by-case” workaround in this situation would be to completely re-work this library to use dynamic require instead of @JSImport purely to accommodate http4s’ cross-building needs, even though that library itself only cares about the Node.js environment. I highly doubt such a disruptive and unergonomic change would or should be accepted.

Cross-compilation is viable when the entire ecosystem is uniformly cross-compiled.

This will likely never happen, since there are many projects that can support both Node.js and Browser environments with exactly the same code (e.g., Cats, Circe). Perhaps http4s really is a fringe case. However, do we truly need uniform cross-compilation across the ecosystem? Consider how you can depend on a library with for3Use2_13 (though much ill advised!! the illustration here is how to adjust the behavior of %%). Similarly, for a crossProject(JVMPlatform, NodeJSPlatform, WebAPIJSPlatform) couldn’t we have a method ("org.typelevel" %%% "cats-core" % "2.6.0").pureJS (or better named) to indicate that there is no NodeJS or WebAPIJS variant and the standard sjs suffix should be used? And also, forJSUseNode or forJSUseWebAPI for a crossProject(JSPlatform) that wants to depend on such crossed upstream projects.

2 Likes