Ambiguous syntax for context functions

Question
1

I’m aware this most likely has been asked before, but I couldn’t find it.

I have the following code, which expresses a very simple random generation capability:

//> using scala 3.8

trait Rand:
  def nextInt(max: Int): Int

object Rand:
  def int(max: Int): Rand ?-> Int = 
    handler ?=> handler.nextInt(max)

  def listOfN[A](n: Int, randA: Rand ?=> A): Rand ?->{randA} List[A] =
    List.fill(n)(randA)

  def apply[A](body: Rand ?=> A): A =
    val rand = scala.util.Random()

    given Rand = max => rand.nextInt(max)

    body

The important bit is how listOfN takes an effectful computation - a Rand ?=> A and not an A, and how eager evaluation of context functions can cause programs to type check but not behave the way they’re intended to at all.

Now, to generate a random list of integers, one merely needs to run:

Rand:
  Rand.listOfN(4, Rand.int(5))

This compiles, runs, and evaluates to something like List(1, 4, 0, 3). No surprise so far, this is exactly what I would expect.

For some reason, I may come back to this later and decide to extract Rand.int(5) - maybe it’s a little too noisy:

Rand:
  val content = Rand.int(5)
  Rand.listOfN(4, content)

With the way context functions are applied, combined with automatic conversion to context functions, this:

  • pulls a random number, let’s say 2, and binds it to content.
  • transforms content in the listOfN call to (rand: Rand) ?=> content - in our example, the effectful computation that returns the constant 2.

We always get random lists of size 4, but that always contain the same element - List(2, 2, 2, 2) in our example.

Of course, this all makes sense if:

  • you have a fairly good grasp on context function and some of their (as of yet undocumented?) features.
  • know all the types and realise that Rand.listOfN takes an effectful computation.

But I would argue that this is not always going to be the case - my gut feeling says it’s more often than not not going to be the case.

I also know you can disambiguate with explicit type ascriptions: val content: Rand ?→ Int = Rand.int(5). But in order to do that, you need to know that the problem exists, and I think the bad part of this problem is that most of the time, you won’t realise this is going on. I didn’t.

Is there some way of working around this, some compiler flag to warn, some bit of syntax I’m not aware of?

1 Like
2

One could make an analogy with by-name arguments, which have pretty much the same issue:

scala> List.fill(4)(util.Random.nextInt(5))
val res2: List[Int] = List(4, 0, 3, 3)
                                                                                                    
scala> val content = util.Random.nextInt(5)
val content: Int = 4
                                                                                                    
scala> List.fill(4)(content)
val res3: List[Int] = List(4, 4, 4, 4)

This is only surprising if you don’t know that List.fill takes a by-name argument.

3 Likes
3

One very well might, and one, indeed, has - why do you think my example includes List.fill? :slight_smile:

I think the point is:

  • you must know all the types, all the time, otherwise you expose yourself to this kind of unpleasantness.
  • it’s kind of ok with by-name, since they’re more of an exception, you don’t encounter them all that often.
  • I expect (hope!) that capabilities will take off and will be used everywhere. This behaviour will become the rule rather than the exception.

If capabilities do take off, this may end up adding a lot of friction / magic to the language…

3 Likes
4

Let me ask a meta-question: if I decide that I’ve had enough of having to lift ‘magic’ behaviour (such as generation of test cases) that I want to thread through a computational flow into monadic types, then what is the state of the art in Scala right now that’s available out of the box?

I’ve not bothered to track the bleeding edge of Scala development, but I’ve been seeing mention of ‘direct style’ / Caprese for a while.

Has the dust settled yet?

EDIT: I saw this ICFP talk, I think the air is still pretty cloudy for now. Carry on as you were…

5

I’ve replied to this in ActivityPub, but I’ll add this here as well:

I think the correct way to refactor code is to extract values into a def. Using val is effectively the same as using IO.memoize in IO-land, so one shouldn’t be surprised that the code is not referential transparent and effects only happen once.

As in

Rand:
  def content = Rand.int(5)
  Rand.listOfN(4, content)

Does the expected thing. I would assume someone would only use val if they explicitly want to memoize the result.

As an aside, I wonder if by-name arguments should be replaced by empty context functions (i.e. () ?=> T). It feels like it would make this situation more regular :thinking:

6

That does fix the problem, yes - in the same way that using a val but with an explicit type ascription would. But my point was in order to use the workaround for the problem, you have to know the problem exists in the first place, and the only way to know it is to know all the types - something I usually lean on the compiler to do for me, because I can’t hold that many things in my head.

Requiring a syntactic distinction between “use this as a value” and “use this as a context function” would solve the problem, at the cost of a little boilerplate, for example.

7

Maybe I’m missing something, but I think this is only true for the type annotation workaround, not for def - which I would argue is not a workaround, but the proper way to refactor the code and ensure RT (either that or inline def).

This is quite different from IO-land, where one can always default to val (as val x: IO[T] behaves somewhat similarly to def x: T).

Refactoring with a def is always safe, it doesn’t matter if it’s a context function or a value. For example, you could refactor the code further with:

Rand:
  def listSize = 4
  def content = Rand.int(5)
  Rand.listOfN(listSize, content)

It doesn’t matter if listSize is a constant value or a context function, the code works as expected.

You only need to know all the types if you want to use val - My point is that val is a performance optimization, so it’s natural that you would need to know those details - as you would if you used IO#memoize or forced a IO#unsafeRunSync in your code to avoid repeating some computations.

8

Oh I see, you’re arguing that in the context of capabilities, we shouldn’t even use the val keyword, ever, just to be sure.

While I can’t think of a reason why this wouldn’t work (meaning I think it works, but am edging my bet because I’ve been wrong before :slight_smile: ), it’s… unpleasant, isn’t it? Also, I think it’s a pretty safe bet this won’t become the norm: it hasn’t for call-by-name, which exhibit the exact same problem.

9

I think context functions are problematic for this use case. Rand is a stateful computation.

I would very much disagree with that. val is perfectly fine (in fact, essential!) with capabiiltes. I think the problem is with your use of context functions. Context functions are fine if the function they describe is referentially transparent. But they become problematic when there are side effects since application of the function is implicit so you don’t see when the side effects happen. Rand => T is definitely not referentially transparent. So I would either use explicit functions or inject the whole thing into a monad. In both cases you have more control what gets applied when.

10

Unless I’m completely mistaken, Rand is not a stateful computation but a capability, isn’t it?

And it was my understanding that stateful computations that work with capabilities, such as, say, Rand.int, were encoded as context functions of shape Rand ?→ A (there is some debate over whether they should only be of that type when passed around, and should be written as defs with givens when declared, but I don’t think that’s very relevant here).

Have I completely misunderstood capabilities?!

1 Like
11

A random generator is a natural capability but Rand => Int is a stateful computation in the sense that each application. will return a different result. I believe it’s better to make these explicit.

Here’s a version of your code that follows this principle:

trait Rand[T]:
  def next(): T

object Rand:
  private val rand = scala.util.Random()

  def int(max: Int): Rand[Int] =
    () => rand.nextInt(max)

  def listOfN[A](n: Int, randA: Rand[A]): Rand[List[A]] =
    () => List.fill(n)(randA.next())

The stateful computation is in this case embedded in the next function of trait Rand. Calling next is even more explicit than applying a function, which IMO is a good thing in this context.

12

-Wunused gives a “free pass” to contextual parameters, but if there were a strict mode, it would warn that the implicit parameter is unused:

Rand.listOfN[Int](4, (using contextual$3: Rand) => content).apply(contextual$2)

Then the most hygienic refactor is not just to use def but:

def content(using Rand) = Rand.int(5)

Is there a virtue in a context function that consumes its parameter exactly once, applying it to a function composition?

Currently, we must choose whether to slap a using clause on everything, including collective extensions, but it’s not obvious whether it matters.

I just happened to re-read the Noel Welsh blog from a couple of years ago, which also illustrates these issues.

The blog relies on the unascribed val context not compiling. But in the wrong lexical context, it becomes what is shown in the OP:

  Print.run:
    val message = Print.println("Hello from direct-style land!")
    message.red // wrong
  Print.run:
    val message: Print[Unit] = Print.println("Hello from direct-style land!")
    message.red
3 Likes
13

I think even there are “right” solutions to the problem the status quo where the language happily accepts a “wrong” solution, which does not what was intended but “looks correct on first sight” (like it’s almost always the case with typical “AI” output!), is quite terrible.

The language should always push the users, even if they don’t know all the fine details (which is to be expected from the majority of users!), to the “right”, safe solution(s).

So I think @som-snytt is here on something.

There should be some kind warning (or similar) when someone “holds it wrong”; a warning which explains what’s possibly a problem, and how the “right” solution in that case looks like. (What’s the “right”, recommencement solution is up to be debated.)

It’s not enough to say that because there are ways to model things properly there is no issue at all, like @Odersky did. Alone that there is a way to arrive at a at first “good looking”, but wrongly modeled solution without the compiler telling you that this is smelly is the core of the issue.

It’s not OK to say that “if you’re holding it wrong it’s on you”. A helpful language tells people when they “hold it wrong”, and what to do instead, ideally with some copy-pastable suggestion in the warning message.

Also such stuff definitely needs in-depth documentation! Lacking documentation is really the bane of current Scala. This is a major problem. (Even “nobody reads docs since ‘AI‘”, this is a problem alone for the reason that “AI” is completely useless without a lot of good training data.)

2 Likes