Super tricky problem; emulating optional parameters. Please help

Have been stuck on this small problem for a few days now. I don’t even know what to title this at this point. Would appreciate help!

Problem: I am building a DSL where this is the notation (and must be the notation)

val foo = Foo(0).modify(
  f = x + 100, // sets foo.f = f
  g = "hello!  " + y // sets foo.g = g
)

// ... later on
foo.modify(
  g = "awesome " + y,
  h = 10 * x
) // original f unchanged

// again later on
foo.modify(
  f =  3 * x
) // most recent g and h remained unchanged

Foo internally stores context lambdas f,g,h, and dozens more, and modify is simply a setter. The parameters for modify need to be optional, as the user may only be needing to re-assign one or a few rather than all of them each time.

Since scala does not have optional parameters, I at first tried using union with Null

case class Foo(id: Int):
  var f: (x: Int) ?=> Int = x + 3,
  var g: (y: String) ?=> String = "ok  " + y
  var h: (x: Double) ?=> Double = y * 10

  def modify(
    f: Null | (x: Int) ?=> Int = null,
    g: Null | (y: String) ?=> String = null,
    h: Null | (x: Double) ?=> Double = null
  ) =
    if f != null then this.f = f
    if g != null then this.g = g
    if h != null then this.h = h

    this

But found that unions of all sorts completely break context functions. Thread here Type a parameter as either a contextual function | Null

Unlucky.

Next I attempt to have the default value of unspecified arguments simply be the original member

 def modify(
    f: (x: Int) ?=> Int = f, // if no new f, use original f
    g: (y: String) ?=> String = g, // if no new g, use original g
    h: (x: Double) ?=> Double = h // if no new h, use original h
  ) =
    this.f = f
    this.g = g
    this.h = h

// override f and g defaults
val foo = Foo(0).modify(
  f = x + 100,
  g = "awesome " + y
) // since h is not specified it's default argument is just the same original h.

// later, override h without resetting previous f,g functions
foo.props(
  h = x * 2
) // defaults this.f = f; // this.g = g

Unfortunately, this is being interpreted as a redefinition of each function pointing to themselves, creating infinite recursion and stack overflow on invocation.

given Int = 1
println(foo.f) // Stack overflow, infinite recursion

Third, I attempted to do a == check between the original and new lambdas to know if a new one has been given first before attempting to reassign.

 def modify(
    f: (x: Int) ?=> Int = f, // if no new f, use original f
    g: (y: String) ?=> String = g, // if no new g, use original g
    h: (x: Double) ?=> Double = h // if no new h, use original h
  ) =
    if f != this.f then this.f = f
    if g != this.g then this.g = g
    if h != this.h then this.h = h
    this

Unfortunately the compiler interprets this as me immediately attempting to invoke f,g,h and compains about no givens found in the scope. 2 threads here:

I did find a hacky workaround, which involves pushing the lambdas into a Seq[(x: Int) ?=> Int] and then indexing seq(0) to retrieve it casted as Any so the compiler doesn’t attempt to invoke it.

val seq = Seq[(x: Int) ?=> Int]
seq = f +: seq
seq.asInstanceOf[Seq[Any]](0) == seq.asInstanceOf[Seq[Any]](0)
// true! f is f!

but immediately found that this == check won’t work anyway as the parameter f and the variable f are unique values on the runtime.

val seq = Seq[(x: Int) ?=> Int]
seq = f +: seq
seq = this.f +: seq
if seq.asInstanceOf[Seq[Any]](0) != seq.asInstanceOf[Seq[Any]](1) 
then // unfortunately this does return true. 
  this.f = f // thus causing infinite recursion here.
// f and this.f are not the same lambda :(

I’m at a lost for what to do next.
@Sporaum suggested to use Option instead of Null

val foo = Foo(0).modify(
  f = Some(x + 5),
  g = Some("awesome" + y)

)

but this obviously destroys the ergonomics of the library. (you will type Some() literally hundreds of times in large projects. A goal of this library is to prove that DSL’s don’t have to look and feel terrible / boilerplatey. This defeats that goal.)

I would be fine with the Option approach if I could create an implicit conversion to avoid having to type Some() on the outside.

Unluckily again, implicit conversions into Some seem not to work for context functions uniquely.

  def modify(
    f: Option[(x: Int) ?=> Int] = None,
    ...
  ) = ...

given Conversion[(x: Int) ?=> Int, Option[(x: Int) ?=> Int]] with
  def apply(f: (x: Int) ?=> Int): Option[(x: Int) ?=> Int] = Some(f)

val foo = Foo().modify(
  f = x + 5 // Not found: x
)

Does anyone have any clever ideas? It is cruciual that the DSL notation shown does not need to change. Getting this exact syntax to work is the entire motivation, please do not recommend an alternative style. I have abstracted away the details of the use case for brevity and don’t want to justify the choice of the syntax, just trust me that this foo.modify(...) notation results in extremely clean domain modeling and anything less will be unsatisfying.

Thanks!

Forgive me if I’m wrong, it seems like you are new to Scala and trying to force ideas / designs you are familiar with from other languages into it. This is highly non-idiomatic Scala. Looks like it is fighting against the language.

It’s not a good idea to design a DSL by saying “must”. If you continue down with this design I think you will suffer a lot, and even if you succeed, the end result will not be good.

In a more idiomatic way, you should design the case class with:

val newFoo = Foo(0).copy(f = x => x + 100 \* , etc. *\)

Apologies if I offended you. Hope it helps.

2 Likes

This is highly non-idiomatic Scala.

Yes because I am not solving an ordinary problem. I’m inventing a sub-language for a specific domain, with scala as the host language. I am not implementing business logic, so my code will not look like idiomatic business logic code. The business logic code I do typically write looks nothing like this. But now I am inventing magic syntax for end-users. And on the outside, for those users, much of the code will be highly idiomatic; one feature of this DSL will be that it is purely immutable (as far as users can see and manipulate). It’s the inside we have to get working (by any means necessary)

it seems like you are new to Scala and trying to force ideas / designs you are familiar with from other languages into it.

This design is not even close to being possible in other languages. I’ve moved to Scala specifically because it can support this DSL. It is one inch away from being a success story.

even if you succeed, the end result will not be good.

The end result will be fantastic. I have already written a few examples and it looks and feels great, everyone I’ve shown (who uses the traditional DSLs for the sub domain) say they love it in comparison. This is just one small kink I need to iron out and the DSL is finished - provide a way to only reassign a a few of the members on each modify rather than all of them each time.

Respectfully, please consider that you don’t know what domain I’m working in, what problem I am trying to solve, and how bad the proposed solution of needing to write .copy(f = x over and over again everywhere in will look like in real project code for this domain. The DSL will lose all appeal to my target audience if I can’t get something like this working. Please trust me that I have thought about the idiomatic alternatives, this DSL is many months in the making. But I appreciate your response.

Maybe the following approach could meet your requirements?

import scala.compiletime.uninitialized

case class Foo(id: Int):
  var f: Int ?=> Int = uninitialized
  var g: String ?=> String = uninitialized
  var h: Double ?=> Double = uninitialized

  def modify(
    f: (Int => Int) | Null = null,
    g: (String => String) | Null = null,
    h: (Double => Double) | Null = null
  ) =
    if f != null then this.f = (x: Int) ?=> f(x)
    if g != null then this.g = (x: String) ?=> g(x)
    if h != null then this.h = (x: Double) ?=> h(x)

    this

// override f and g defaults
val foo = Foo(0).modify(
  f = _ + 100,
  g = "awesome " + _
) // since h is not specified it's default argument is just the same original h.

// later, override h without resetting previous f,g functions
foo.modify(
  h = _ * 2
) // defaults this.f = f; this.g = g

// test it
foo.f(using 42) // 142
foo.g(using "DSL") // awesome DSL
foo.h(using 1.0) // 2.0

Unfortunately each contextual function actually has multiple parameters, not just one, so _ likely fails.

foo.modify(
  f = 3 + x + someothercontext.n + y
)

so f here has x, someothercontext, and y all three as contextual parameters.

And it’s also just important for the readability of the DSL that the parameters are named.

It’s also possible to use multiple or named parameters:

import scala.compiletime.uninitialized

case class Foo(id: Int):
  var f: (x: Int, y: String) ?=> String = uninitialized
  var g: (x: String) ?=> String = uninitialized
  var h: (x: Double) ?=> Double = uninitialized

  def modify(
    f: ((x: Int, y: String) => String) | Null = null,
    g: ((x: String) => String) | Null = null,
    h: ((x: Double) => Double) | Null = null
  ) =
    if f != null then this.f = (x: Int, y: String) ?=> f(x, y)
    if g != null then this.g = (x: String) ?=> g(x)
    if h != null then this.h = (x: Double) ?=> h(x)

    this

// override f and g defaults
val foo = Foo(0).modify(
  f = (x, y) => s"$y: $x",
  g = x => "awesome " + x
) // since h is not specified it's default argument is just the same original h.

// later, override h without resetting previous f,g functions
foo.modify(
  h = x => (x + 1) * (x + 2)
) // defaults this.f = f; this.g = g

// test it
foo.f(using 42, "result") // result: 42
foo.g(using "DSL") // awesome DSL
foo.h(using 1.0) // 6.0

Sorry, needing to explicitly write the lambda parameters first hurts the DSL appeal too much. I appreciate the solution though. I have to find a way to keep using context functions. Just need a way to represent “do nothing if this argument is unspecified”, crazy that this is so impossible to do.

If Scala provided any real as Any way to tell the compiler “shut up, pretend what I’m about to give you is T, do no attempt any verification in the runtime”, I could simply do this

object IsDefault

  def modify(
    f: (x: Int ?=> Int) = IsDefault.asInstanceOf[(x: Int) ?=> Int],
  ) = 
    if f.asInstanceOf[Unset.type] != Unset then
      this.f = f

But unfortunately .asInstanceOf does actual runtime stuff, actually changing the value of the object where the result after the cast is literally not the same object. So crazy.

Sure, this is bad code that should be replaced with something like a Union. But Scala context functions conveniently don’t work with Unions at all, just my luck.

If I understand your goal correctly of how you want to call modify, there might be another thing to consider: you cannot implicitly access the name of a context function parameter.

I.e. the following on its own will not compile:

val f: (x: String) ?=> String = x // Error: Not found: x

In order to define f in this way, you would have to retrieve x from the implicit scope with a helper function, like so:

val f: (x: String) ?=> String = x // ok

def x(using String) = summon[String]

Now this may or may not be what you are looking for, just a heads up :slight_smile:

Actually it does work as of 3.5 I believe.

1 Like

You are right, I was on the LTS. Thanks for the tip!

I must underscore this, I would start by making a regular scala API, and only then try to make a fancy DSL for this has a couple upsides:

  • Helps you get familiar with basic scala
  • Helps you iterate faster on your design (DSL are much harder to evolve than APIs)

Sure, fair advice in general. My project did start as a regular scala API, and it was no improvement over the DSLs in the mainstream ecosystems, other than the fact that now we get to use Scala which means little to my coworkers. The project has gradually turned into this fancy DSL as I’ve worked to get rid of all the rough edges me and my coworkers hate about the available DSLs, by combining all of the advanced language features of Scala to get a magical looking sub-language working (implicits, contextual functions, infix notation, symbol operators, opaque types, etc.). Now I have a DSL that feels far better than the traditional options in other languages, some of the people I’ve shown the real intro examples to say they are willing to switch to Scala to use it.

I would underscore that this is a sort of language I’m building, not just a library. It’s a Scala-based DSL meant to replace pre-existing markup languages that look nothing like Scala, so that me and my coworkers can do all of our work in one real programming language instead of spread across multiple file types and semantics. So the drift away from idiomatic scala library API and towards crazy black-magic in some areas is expected.

I know these abstracted snippets I’ve posted for brevity look strange, (I’m not even using the same words as my real use case uses), but there is a very valuable vision motivating it you’ll just have to trust is there, and it will result in an excellent experience by the end if I can just finish this last bit of work.

Okay, this is quite a stretch.

The primary problem is just that the compiler is very eager to apply the given. (Note that this is all using Scala 3.6.4.)

scala> val f: (x: Int) ?=> Int = 3 + x
val f: (x: Int) ?=> Int = Lambda/0x0000785ab867d770@2962ddba
                                                                                
scala> val g: AnyRef = f
-- [E172] Type Error: ----------------------------------------------------------
1 |val g: AnyRef = f
  |                 ^
  | No given instance of type Int was found for parameter of (x: Int) ?=> Int
1 error found

This is not because context functions are not AnyRef! They most certainly are.

scala> abstract class AsAnyRef() {
     |   def f: AnyRef
     |   final def anyref_f: AnyRef = f
     | }
// defined class AsAnyRef
                                                                                
scala> class IsF() extends AsAnyRef() {
     |   val f: (x: Int) ?=> Int = x + 3
     | }
// defined class IsF
                                                                                
scala> val isf = IsF()
val isf: IsF = IsF@3ceb5831
                                                                                
scala> val f = isf.anyref_f
val f: AnyRef = IsF$$Lambda/0x0000785ab867e740@fd5b66d
                                                                                
scala> val fx = isf.f(using 5)
val fx: Int = 8

So, we have the capacity to get these things as AnyRef. It’s just a pain–we’ve had to trick it with subtyping. (You can assign it to a variable of the exact right type, but if you try to assign it to a supertype, it balks and tries to apply the given instead.)

Okay, now the question is whether we can detect a default parameter.

To make a very long story short, the answer is yes, but because the compiler really, really, really wants to apply the given whenever possible, we have to go to quite some lengths to trick it.

The first thing to notice is that if you have a lambda that evaluates context function, e.g. () => f(given 0), we can put in a dummy value to fulfill the given there and never need to actually call it. Even inline foo(inline arg: Any) will try to fulfill the given rather than passing it as arg, but we can use () => ... to capture the given whole by promising we’ll apply it when we call it.

The second thing to notice is that context functions are also very eager to regenerate themselves, which means that you can’t really get a stable memory reference to them. Even if you use the anyref_f trick above, you’ll find that patterns that would result in a stable function value do not result in a stable context function; there’s an extra layer of indirection via invokedynamic.

But all is not lost! The expressions are still different, so we can potentially get a macro to do the work the work of determining whether an expression is valid or not. We just have to make sure we apply the macro to an expression that captures the context function rather than trying to apply the context.

So, here are the key pieces.

We need modify to be inline and all of the arguments of the type f: (x: Int) ?=> Int = default to be inline, so that we can inspect the expressions with a macro.

We can’t inspect f directly, but rather have to inspect (inline) () => f(using 0) to keep the eager usage of givens happy but not actually do anything. What we look for is the presence of default, which is a bit fragile but rather than calling it default we can name it something that the user really really should not be using on their own directly like __do_not_use__.

Then we have to traverse the tree of expressions inside a macro and determine (statically) whether or not we should actually save the lambda. Ideally we’d simply parse out the (inlined) default argument. But that kind of thing always is rather shaky in my hands; small differences sometimes result in the tree being formed differently and then the find-the-exact-structure approach breaks. So instead we traverse the entire tree and look for any instances of the default identifier. Furthermore, if we want to nest modifies, we’ll need to avoid traversing the parts of the tree that have those. So the key bits of the tree traversal will have to look something like

term match
  case Ident(name) if name == "__do_not_use__" => true
  case Apply(Ident(name), _) if name == "modify" => false
  case Select(_, name) if name == "__do_not_use__" => true
  case Select(_, name) if name == "modify" => false
  case _ => ??? // Recurse over tree

Putting this all together is a challenge. I have a working draft up here: Macro solution to context function default argument evaluation · GitHub

You can run it with scala-cli Macros.scala Main.scala if you copy the files somewhere.

(You can set the inline val debug to true to print out expressions in case not all the pieces work, but scala-cli eats compiler output; not sure whether you can turn that feature off, but if not you’d have to use something else to get debug output.)

I can’t make it runnable online because Scastie doesn’t allow multiple files as far as I know, and macros require them. (You can paste in the REPL, though.)

With this you can do things like

    val foo = Foo(3)
    val fue = Foo(8)
    foo.modify(
      g = y + y,
      h = x + { fue.modify(f = x + 5); fue.f(using 3) }
    )

and it all works.

(I made modify return Unit but you can type it as this.type and add a this on the end if you prefer.)

3 Likes

The fact that this one does not work (and especially that it leads to SO at runtime) makes me think I should never use contextual functions.

1 Like

This is because context functions are eager to apply their argument rather than be passed around as values. So you create a new context function that gives its context to whatever is stored in the var. But what do you put in that var? The same context function! So of course you get a loop.

It acts like

class Oops:
  var thunk: () => Int = () => 2
  def modify(think: () => Int = () => thunk()):
    thunk = think

If you could write think: () => Int = thunk, then it would work; you wouldn’t create a loop. But because Scala interprets A ?=> B = f with f: A ?=> B as (a: A) ?=> (f(using a): B), you get a loop.

Right, and it really should not do that IMO. Even in simpler use cases that don’t break down because of this, at the very least it will create a whole bunch of unnecessary indirection.

1 Like

This might all be reasonable and expected once one understood the overall mechanism. But for me, it’s far too much cognitive load for anyone who needs to understand my code compared to the benefits of contextual functions.

I’d hope that the compiler would at least issue a warning if one ends up implementing endless recursions in that way (just like it does in some other cases of endless recursion loops).

1 Like

IMHO the root issue is just the over focus on making the syntax lightweight.
As I said before, this same issue happens with by-name parameters.

For those, I proposed that Scala 3 should have changed them to just be a () => A inside the method, meaning that you would need to do f() in order to invoke it, but users would still get the improved syntax of not having to pass the () =>.
But that proposal was rejected because it was “too verbose” and rather Martin wanted to complicate it even further (curiously using context functions).

For context functions I would propose something similar. The implicitness of the parameter should be restricted to the call site. But, once you have a value of it, it should be just another normal function.

1 Like