Http clients for Scala?

What are good http clients for Scala?

At work we’ve been working around a bug in async-http-client for some years and I’ve finally gotten permission to replace it. (The bug is at the boundary between async-http-client and netty . Characteristic symptom is Caused by: io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultHttpRequest, state: 1 , described at Duplicate handler name: request-body-streamer · Issue #1660 · AsyncHttpClient/async-http-client · GitHub . Retrying works around the problem, but I explain what is happening too frequently.)

We have several hospitals who use client-side http proxying - that’s likely the most stringent requirement. (That unfortunately eliminates the Blaze client - we’ve been very happy with every other aspect of http4s.)

We’d very much prefer something NIO-based, or at least non-blocking. A client that can recycle existing connections would benefit performance. Something that plays nice with http4s already would save us work.

What’s out there for us to try?

Thanks,

David

Have you checked out sttp? It supports multiple backends.

2 Likes

Thanks, Will.

My main concern with a front-end facade/back-end plumbing approach is it makes our project beholden to at least two more communities. I’d prefer just one.

One advantage of using the Blaze client was that we are already dependent on the http4s community. When we switched to the http4s wrapper around async-http-client (and netty) we were dependent on three communities. The http4s community (Ross Baker I think) brought it to their attention - even tried to fix it. Neither of the other two communities wanted to own the problem.

sttp does provide a list of http clients to try out. That’s really useful.

Haven’t tried it but maybe?

GitHub - com-lihaoyi/requests-scala: A Scala port of the popular Python Requests HTTP client: flexible, intuitive, and straightforward to use.

2 Likes

I will note that our stack at work (probably the better part of a thousand microservices talking to each other) is mainly based on play-ws. Not sure whether I would recommend that for a non-Play project, and I don’t know how well it copes with your proxy requirement, but it’s done decently well by us AFAIK.

1 Like

Again, http4s has a client too. It shared API with the server.

See e.g. https://http4s.org/v0.19/client/

1 Like

Thanks Hmf. That library is definitely worth considering, even though we’re not too concerned about its “easy to do one-off requests” focus. We’ll have to bind whatever we choose to cats io effects.

There’s a real treasure trove of options - clearly not Li Haoyi’s intent - at GitHub - com-lihaoyi/requests-scala: A Scala port of the popular Python Requests HTTP client: flexible, intuitive, and straightforward to use. .

Thanks Naftoli.

I’ve decided to evaluate adding client-side http proxying to Blaze (http4s’ preferred client). That would be a very clean long-term fix, and closer to my ideals. It is a much bigger up-front effort than adding a new dependency, with a longer lead time before we can release. That makes it harder to convince the rest of the team.

Thanks Mark. I’ll evaluate it, see what it would entail. Last time we looked (2018?) Akka’s client didn’t support the client-side proxy feature. We’re not yet dependent on Play; if it starts to pull in the full framework (and Akka) then it’s likely not the right choice for us.

Does anyone have experience (positive or negative) with JDK11’s new HttpClient (Java SE 11 & JDK 11 ) ?

(It has a sendAsync method HttpClient (Java SE 11 & JDK 11 ) and at least promises to support the client-side proxy.)

If this works it’d be a tidy conclusion: http4s-jdk-http-client

It shouldn’t pull in the whole framework (modern Play is at least adequately decoupled), but probably does pull in Akka.

Taking a quick glance at that, I just note that the documentation looks to be slightly out of date: the build.sbt is pointing to cats-effect 3, but the doc example refers to ContextShift. It’s probably fine, but keep that in mind if you hit slight inconsistencies. (Not astonishing, given that CE3 dropped so recently.)

2 Likes

There is also akka-http client as an option.

Lately though I prefer to just use sttp.

1 Like

Thanks all for the help with this one. The http4s-jdk-http-client worked really well for us, at scale, and in two proxy configurations.

1 Like