I’m afraid this question is very generic (and fuzzy) and very specific at the same time, but perhaps somebody here has already faced a similar scenario and can offer some insights…
In a nutshell, I want to create a REST service that accepts POST requests with JSON payloads and push these payloads to a Kafka topic. The stack used so far is cats/cats-effect, fs2 and http4s. I have a basic understanding of Kafka (I hope), but no in-depth hands-on experience, yet.
Ideally, I’d like to address the following (partially conflicting) concerns:
- Request/response round trip time should be as quick as possible.
- The response should be issued as late as possible in the delivery process, so I can notify the client with a 500 if message delivery failed at some point.
- Server should not crash under load.
- No pending messages should get lost if the server crashes nevertheless.
- Server should start refusing (some?) requests with 503 right at the entrance when it can’t keep up with pushing incoming messages to Kafka.
(I’m aware I probably can’t have it all, that’s just the ideal…)
So far I have looked into kafka4s and fs2-kafka. Both are backed by the Apache Java client.
I have a naive implementation using kafka4s calling ProducerApi#sendAsync() inside my HttpRoutes. I still need to wrap my head around the various timeout/retry/ack settings in the Kafka client config, but basically this seems to work. I just have no idea how I should accomplish backpressure handling from here.
As an alternative, I’ve tinkered with fs2-kafka. Here the first challenge is to get the single requests from the HttpRoutes “callback” into a Stream. The best approach I’ve found so far is going through an internal queue. This sounds somewhat undesirable, as it introduces pending local state (in addition to the Java client’s buffer) that will be lost without a trace on a server crash. OTOH I have a feeling that using a bounded queue here might somehow give me a handle on backpressure, but I don’t have a clear idea, yet.
Has anybody here already done something like this? Is there any example code out there going from http4s requests to Kafka (or some similar scenario) that I could use for inspiration? Any hints, not necessarily restricted to the stack/libs mentioned above, are appreciated.