As I’ve said, the problem is that Scala 2.11 compiler closes over the function T => Future[S] unnecessarily in the inner Future.onComplete. Viktor Klang wrongly attributed the improvements to his promise linking rather that to fixed lambda encoding in Scala 2.12.
To see the problem you need to disassemble (using javap -c) some classes from Scala 2.11 library. Download and unpack https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.tgz . After looking at scala/concurrent/Future.class and it’s inner classes you’ll see that:
- closure passed to onComplete at line 251 closes over parameter
f: T => Future[S] and local variable val p = new DefaultPromise[S]()
- closure passed to onComplete at line 256 closes over previous closure to be able to access promise
p, but that also means it closes over parameter f because previous closure closes over it
Workaround for that compiler bug is here: https://github.com/tarsa/FuturesFix/blob/ce7030153557370095441ad470c8c003d5751fe1/src/main/scala/futures/fixed/concurrent/Future.scala
There’s a function factory method that avoid unnecessarily closing over unwanted variables (i.e. it closes over only the provided parameter):
private def feedPromise[S](promise: Promise[S]) =
promise.complete(_: Try[S])
I’m using it then in flatMap in a way that avoids mentioned unwanted closure:
def flatMap[S](f: T => Future[S])(implicit executor: ExecutionContext): Future[S] = {
val p = Promise[S]()
onComplete {
case f: Failure[_] => p complete f.asInstanceOf[Failure[S]]
case Success(v) => try f(v) onComplete feedPromise(p) catch { case NonFatal(t) => p failure t }
}
p.future
}
Now let’s return to see in which cases promise linking can help. Consider following loop:
val start = Promise()
var previous = start
while (true) {
val current = Promise()
current.completeWith(previous.future)
previous = current
}
Above code effectively build infinite chain of promises. Without promise linking we’ll end up with infinite chain of callbacks. With promise linking we’ll end up with infinite chain of promise links. But together with promise linking there’s additional heuristics that tries to shorten paths whenever possible. At any iteration of the loop except first, promise previous is a link to some previous promise. When linking additional promise (current) it is possible to follow the promise links up to a promise that is not a link (it’s called a root promise). Then instead of linking to immediate predecessor we can link to root. Heuristics also relinks all promises on the path to root, not only the current promise, to make all paths as short as possible. Unfortunately this heuristics doesn’t reliably compress all paths in all cases so relinking is done all the time, hoping for good enough real world guarantees.
A much less contrived example is for comprehension on Futures (or equivalent flatMap/ map/ filter sequence). If we have N flatMaps in a row (and no promise linking) then we need to keep N + 1 Promises in memory. With promise relinking we can reduce the number of reachable (by GC) Promises to 2 or 3. But in practice the gain is small - usually for comprehensions on Future’s aren’t super long. Every Promise takes maybe a dozen or few dozens of bytes, so even if there are 20 Promises in a chain inside for-comprehension then it translates to maybe a kilobyte of saved memory.
In general, I think that the whole promise relinking looks like over-engineering catered at very special use cases. Promise relinking heuristic does not give 100% guarantee that all paths will be compressed so every weird use case like ACPS needs to be tested against promise relinking heuristic to find out whether it’s sufficient.
