Idiomatic way of tail-recursive function returning Future

Question
1

I need to download data from a service that gives data under page-cursor pattern:

@tailrec
def download(nextPageCursor: Option[String], result: Future[Seq[T]]): Future[Seq[T]] =
  val responseFut: Future[(String, Seq[T])] = restApiCall(nextPageCursor)
  responseFut.flatMap { (nextPageCursorNew, data) =>
    val resultNew = result ++ data
    if nextPageCursorNew.isEmpty then resultNew
    else download(Some(nextPageCursorNew), resultNew)
  }

val allData = download(None, Future.successful(Seq.empty))

This doesn’t work since download is not in tailrec position. How to make it?

I found trampoline pattern. Is that the idiomatic way? But I find it verbose. Is there a library (e.g. cats probably) with implemented abstractions for the pattern?

2

You don’t need tailrec with Future, just remove that.

Future.flatMap is stack safe, which is all that matters.

2 Likes
3

Huh – I wonder if that changed somewhere along the line. I would swear I had to implement a crude trampolining in Querki, many years ago in 2.11, in order to deal with stack overflows in some of my Future-based code there.

To the original question: yes, Cats exposes a Trampoline abstraction. I can’t say I’ve ever used it directly – in the Typelevel world, you tend to use IO for this sort of thing, and that’s stack-safe – but in theory it’s the tool you’re asking for. But it sounds like it’s probably unnecessary in this case.

More generally, it’s worth keeping in mind that modern data structures like these are often reasonably stack-smart, so it tends to be a non-issue, especially for async functions.

2 Likes
4

So according to this: Future's Flatmap is not stacksafe · Issue #11256 · scala/bug · GitHub

It will be stack safe if the ExecutionContext is.
And IIUC, the default ExecutionContext should be stack safe.

But, I could be wrong.
I do admit I was too quick to answer based on my knowledge of IO.

2 Likes
5

I’ve made heavy use Future.flatMap recursively (both in JVM and Native) and can say that yes, the default ExecutionContexts are stack safe.

3 Likes