Dotty/ Scala3 Macros: how to unpack a compound type

Question
1

I am trying to implement a concatof 2 HLists. In my first attempt:

  private def unsealUnderlying[T](p:Expr[T])(using t:Type[T], qctx:QuoteContext): Expr[T] = {
    import qctx.reflect._

    val xTree0: Term = p.unseal
    xTree0 match {
      case Inlined(_, _, _) =>  // (Inlined(None, Nil, Ident("fta4")))
        // passed as an identity, so get underlying value
        p.unseal.underlying.seal.cast[T]
      case _ =>
        report.throwError(s"unsealUnderlying: Expected HList reference but got ${p.show} -- (${p.unseal.showExtractors})", p)
    }
  }


  sealed trait HList
  case class HCons[+HD, TL <: HList](hd: HD, tl: TL) extends HList
  case object HNil extends HList

  private def concatImpl[Xs <: HList, Ys <: HList, Zs <: HList](e: Expr[Xs], acc: Expr[Ys])
                                                   (using xt:Type[Xs], yt:Type[Ys], qctx:QuoteContext):
  Expr[Any] = {
    import qctx.reflect._

    e match {
      case '{HCons($a:$t, HNil)} =>
        '{HCons($a:t,$acc)}

      case '{HCons($a:$t,$tl)} =>
        val ntl = concatImpl(tl,acc)
        '{HCons($a:t, ${ntl} )}

      case p:Expr[Xs] =>
        // passed as an identity, so get underlying value
        val expOk = unsealUnderlying[Xs](p)
        // Now deconstruct the value
        concatImpl(expOk, acc)
    }
  }

  transparent inline def concatM[Xs <: HList, Ys <: HList](inline xs: Xs, inline ys: Ys): Any = {
    ${ concatImpl('xs, 'ys) }
  }

I get the error in the second case:

[error] 454 |        '{HCons($a:t, ${ntl} )}
[error]     |                        ^^^
[error]     |                        Found:    (ntl : quoted.Expr[Any])
[error]     |                        Required: quoted.Expr[automl.Recorder.HList]
[error] one error found

If I return a type HList I loose the type information of the HList elements. So how can I extract the type of the remainder of the HList? I then tried to extract and use the type directly from the macro so:

  private def concatImpl[Xs <: HList, Ys <: HList, Zs <: HList](e: Expr[Xs], acc: Expr[Ys])
                                                   (using xt:Type[Xs], yt:Type[Ys], qctx:QuoteContext):
  Expr[Any] = {
    import qctx.reflect._

    e match {
      case '{HCons($a:$t, HNil)} =>
        '{HCons($a:t,$acc)}

      case '{HCons($a:$t,$tl:$tlt)} =>
        val ntl = concatImpl(tl.asInstanceOf[Expr[HList]],acc)
        '{HCons($a:t, ${ntl} )}

      case p:Expr[Xs] =>
        // passed as an identity, so get underlying value
        val expOk = unsealUnderlying[Xs](p)
        // Now deconstruct the value
        concatImpl(expOk, acc)
    }
  }

Ok, so I suspect that asInstanceOf may not be the way to go, however I now get the error:

[error] -- [E007] Type Mismatch Error: /RecordMacro.scala:439:25 
[error] 439 |      case '{HCons($a:$t,$tl:$tlt)} =>
[error]     |                         ^^^^^^^^
[error]     |                         Found:    ev$7.Underlying
[error]     |                         Required: automl.Recorder.HList
[error] -- [E007] Type Mismatch Error: RecordMacro.scala:454:24 
[error] 454 |        '{HCons($a:t, ${ntl} )}
[error]     |                        ^^^
[error]     |                        Found:    (ntl : quoted.Expr[Any])
[error]     |                        Required: quoted.Expr[automl.Recorder.HList]
[error] two errors found

So now I see that the Hlist tail must have it internals extracted. So now I make another try:

  private def concatImpl[Xs <: HList, Ys <: HList, Zs <: HList](e: Expr[Xs], acc: Expr[Ys])
                                                   (using xt:Type[Xs], yt:Type[Ys], qctx:QuoteContext):
  Expr[Any] = {
    import qctx.reflect._

    e match {
      case '{HCons($a:$t, HNil)} =>
        '{HCons($a:t,$acc)}

      case '{HCons($a:$t,$tl)} =>
        val xTree0: Term = tl.unseal
        val ntl = xTree0 match {
        case Inlined(_, _, _) =>  // (Inlined(None, Nil, Ident("fta4")))
          // passed as an identity, so get underlying value
          tl.unseal.underlying.seal.cast[HList]
        case _ =>
          report.throwError(s"unsealUnderlying: Expected HList reference but got (${tl.unseal.showExtractors})", tl)
        }
        '{HCons($a:t, ${ntl} )}

      case p:Expr[Xs] =>
        // passed as an identity, so get underlying value
        val expOk = unsealUnderlying[Xs](p)
        // Now deconstruct the value
        concatImpl(expOk, acc)
    }
  }

and I get the expected error:

[error]     |unsealUnderlying: Expected HList reference but got (Apply(TypeApply(Select(Ident("HCons"), "apply"), List(Inferred(), Inferred())), List(Apply(TypeApply(Select(Ident("Tuple2"), "apply"), List(Inferred(), Inferred())), List(Literal(Constant.String("2")), Literal(Constant.Int(2)))), Apply(TypeApply(Select(Ident("HCons"), "apply"), List(Inferred(), Inferred())), List(Apply(TypeApply(Select(Ident("Tuple2"), "apply"), List(Inferred(), Inferred())), List(Literal(Constant.String("3")), Literal(Constant.Double(3.0d)))), Ident("HNil"))))))
[error]     | This location contains code that was inlined from HyperParameters.scala:314

Now I see that I have the expected type, but how can I use this in the cast? If I use:

          tl.unseal.underlying.seal.cast[HList]

I again end up with a generic HList. How can I use this information to cast to the proper type?

Appreciate any help. Any pointers to examples or documentation is welcome. I have looked at the TASTy documentation but could not figure this out.

TIA

2

Is the issue not that in val ntl = concatImpl(tl,acc) concatImpl has return type Expr[Any] while I suspect it should be Expr[HList]?

3

That is the reported error but is not quite the issue. The goal is to use a transparent inline macro in order to return a specific type. I assume in this case I must use the Any return as shown in the previous link.

If I use the return Expr[HList] then this will compile:

    val hl3 = HCons(("1","1"), HCons(("2",2), HCons(("3",3.0),HNil)))
    val hl4 = HCons(("4",'A'), HNil)
    val rhl4 = concatM(HCons(("1","1"), HCons(("2",2), HNil)), HCons(("3",3.0),HNil) )

But this won’t:

    val hl3 = HCons(("1","1"), HCons(("2",2), HCons(("3",3.0),HNil)))
    val hl4 = HCons(("4",'A'), HNil)
    val rhl4: HCons[(String, String),
      HCons[(String, Int),
        HCons[(String, Double),
          HCons[(String, Char), automl.Recorder.HNil.type]
        ]
      ]
    ] = concatM(hl3, hl4)

It fails with:

[error] -- [E007] Type Mismatch Error: HyperParameters.scala:354:15 
[error] 354 |    ] = concatM(hl3, hl4)
[error]     |        ^^^^^^^^^^^^^^^^^
[error]     |Found:    automl.Recorder.HCons[(String, String), automl.Recorder.HList]
[error]     |Required: automl.Recorder.HCons[(String, String), 
[error]     |  automl.Recorder.HCons[(String, Int), 
[error]     |    automl.Recorder.HCons[(String, Double), 
[error]     |      automl.Recorder.HCons[(String, Char), automl.Recorder.HNil.type]
[error]     |    ]
[error]     |  ]
[error]     |]
[error] one error found

I have successfully implemented a reverse (see below) using this approach. I basically took that and tried to adapt it. Note that I have already tried returning Expr[HList] and even Expr[_ <: HList] with no success.

So I figured, I should somehow extract the HList's tail type and cast the input for the next recursive call to that type.

Maybe I am just looking at this the wrong way, so any corrections and suggestions are welcome.

TIA

  private def reverseImpl[Xs <: HList, Ys <: HList](e: Expr[Xs], acc: Expr[Ys])
                                                 (using xt:Type[Xs], yt:Type[Ys], qctx:QuoteContext):
  Expr[Any] = {
    import qctx.reflect._

    e match {
      case '{HCons($a:$t, HNil)} =>
        '{HCons($a:t,$acc)}

      case '{HCons($a:$t,$tl)} =>
        reverseImpl(tl, '{HCons($a:t,$acc)})

      case p:Expr[Xs] =>
        val expOk = unsealUnderlying[Xs](p)
        reverseImpl(expOk, acc)
    }
  }

  transparent inline def reverseM[Xs <: HList](inline xs: Xs): Any = {
    ${reverseImpl('xs, 'HNil)}
  }

    val t1: HCons[(String, Double),
              HCons[(String, Int),
                HCons[(String, String), HNil.type]
        ]
      ] = reverseM(hl3)
4

IIUC transparent will make sure that user code sees the most specific type. You don’t need to widen your return type to Any for that. If your return type will always be a subtype of Hlist it’s safe to use HList as return type.

5

That’s the issue. It does not seem to do that. As I said, if I return an Expr[HList] or even a Expr[ <: HList] I get:

HCons[(String, String), automl.Recorder.HList]

So the tail is a generic HList.

Maybe I am missing something obvious here. 8-(

6

If returning Expr[HList] vs Expr[Any] makes a difference to how transparent behaves then that sounds like a bug.

7

Ok. I will report this and show both examples. Appreciate the feedback.

8

Small update: the reverse example above works with Expr[HList] and Expr[_<:HList] so, as pointed out to me, I need not use the Any (widen the type). I think the issue is with “extracting” the tail’s full type. Using TASTy shows the information is their, but I cannot seem to put it to use.

I have reported this here but am in doubt if this is a bug.