Functions-remote: a codegen and library to create and invoke functions remotely with customizable serialization (json and Avro supported) and customizable transports (http4s, kafka and anything that can transfer an array[byte] supported)

Announce
1

For scala, the most important ability of the language is the ability to call a function f(x) = y and get its results. And while we can do that within a jvm instance, it would be useful to be able to call functions this easily across jvms that are running in the same box or remote boxes.

Functions-remote is a code generator for calling functions “remotely”, using different serialization methods (like json or avro), and different remote transports (like http). Remotely means we may use http as a transport (i.e. via http4s), kafka where calling is publishing and subscribing is invoking the actual implementation of the function, or just use an isolated classloader as transport so that we can execute the function locally or maybe just spawn a new jvm to call the function.

Effectively functions-remote allows the simplicity of f(x) = y no matter where f will really run.

The generated code is very readable and as if written by a person.

There are many benefits using functions instead of the usual manual serialization of case classes. Apart not having to write any serialization code, it is also a lot more readable and easier to mock in tests. Also easier to change where/how the call will be done in the future if there is a need to do that.

For scala 3. http4s, kafka and an isolated class loader are supported as transports at the moment. Avro & Json as serializers.

Some quick examples:

Http4s

Lets say we have this function:

def ls(path: String)(lsOptions: LsOptions = LsOptions.Defaults): F[LsResult]

Functions-remote will create routes for the server and code for the client to convert the call to an http request to http://server/configurable_path/$path . The http-method is also configurable. And with configurable serialization which at the moment can be json, Avro or both together. This means the http4s server can support both Avro and json serialization.

Calling the function will use http4s client to actually do an http request but the code is simply:

      val f = LsFunctionsCallerFactory.newHttp4sJsonLsFunctions(client, uri"http://localhost:8080")
      val results = f.ls("/tmp/some-dir1")()

on the server side we can add the routes for say both Avro and json:

val imps         = new LsFunctionsImpl
val routesJson   = LsFunctionsReceiverFactory.newJsonLsFunctionsRoutes(impl)
val routesAvro   = LsFunctionsReceiverFactory.newAvroLsFunctionsRoutes(impl)
val routes       = HttpRoutes.of(routesJson.allRoutes orElse routesAvro.allRoutes)

Kafka

Lets say we have this function:

def addPerson(key: PersonKey)(ttl: Long, person: Person): Unit

Functions-remote will convert the call to a publisher for kafka with the Kafka-key being PersonKey and ttl & person as the published data. On the subscriber side, any received data for these calls will be converted to a call to the implementation of the method.

Again both publisher and subscriber can support Avro, json or both at the same time.

Some quick sample code for the producer:

val avroFunctions = KafkaFunctionsCallerFactory.newAvroKafkaFunctions(transport.transportFunction)
avroFunctions.addPerson(PersonKey("person1"))(2000, Person(5, "person-avro")) // sends the data to kafka

Generic

Function-remote also creates generic code where i.e.

def ls(path: String, lsOptions: LsOptions = LsOptions.Defaults): LsResult

Is converted to a call to a transport function (effectively a function of array[bytes] => array[bytes]) and also helps receiving the data and calling the actual implementation.

Functions-remote GitHub page with docs & example code:

2 Likes