//I am trying to dynamically define , compile and access case classes that are //nested
//
package org.example
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox
object DynamicCompilation {
def compileCaseClasses(code: String): Option[ClassLoader] = {
try {
val tb = currentMirror.mkToolBox()
val compiled = tb.compile(tb.parse(code))
// Get class loader from the compiled object
val classLoader = compiled.getClass.getClassLoader
// Print the class loader information
println(s"Class loader being compiled is : $classLoader")
Some(classLoader)
} catch {
case e: Throwable =>
println(s"Compilation Error: ${e.getMessage}“)
e.printStackTrace()
None
}
}
def getClassByName(classLoader: ClassLoader, className: String): Option[Class[_]] = {
try {
Some(classLoader.loadClass(s"org.example.$className”)) // Assuming package org.example
} catch {
case e: ClassNotFoundException =>
println(s"Class org.example.$className not found: ${e.getMessage}")
None
}
}
def main(args: Array[String]): Unit = {
val code = “”"
{
case class Address(
city: Option[String],
country: Option[String],
street: Option[String]
)
case class Person(
address: Option[Address],
age: Option[Long],
name: Option[String]
)
case class Root(
people: Option[Seq[Person]]
)
}
“”"
compileCaseClasses(code) match {
case Some(classLoader) =>
getClassByName(classLoader, "org.example.Address") match {
case Some(addressClass) =>
println(s"Successfully compiled Address class: $addressClass")
case None => println("Could not find Address class.")
}
getClassByName(classLoader, "Person") match {
case Some(personClass) =>
println(s"Successfully compiled Person class: $personClass")
// ... (Example of creating an instance and accessing fields)
case None => println("Could not find Person class.")
}
getClassByName(classLoader, "Root") match {
case Some(rootClass) =>
println(s"Successfully compiled Root class: $rootClass")
case None => println("Could not find Root class.")
}
case None =>
println("Compilation failed.")
}
}
}
May I ask what your use case is? This sort of dynamic compilation is unusual in the Scala world, and access is inherently going to be challenging, since the types don’t exist at the primary compile time.
That said: what’s the question? It’s not obvious what’s going on here, or what problems you’re hitting.
I want to access the case classes I just compiled on the fly. especially if I used one of them to apply type safety to a dataframe to create a dataset in spark
val ds = df.as[Address]
Wouldn’t macros make more sense for your use case? There is also a Scala 3 library that, by my understanding does a similar thing GitHub - VirtusLab/iskra: Typesafe wrapper for Apache Spark DataFrame API
As for dynamic compilation we do that for mdoc but it’s not possible to get those compiled types reliably really, but rather have them behind a trait. Alternatively, you could potentially add those compiled classes to filesystem and the classpath, but that might break easily
1 Like
If I’m understanding what you’re trying to do (which I might not), it’s impossible. If, say, Address is only defined at runtime, then
val ds = df.as[Address]
can’t compile, since Address doesn’t exist at compile-time.
(That’s not a Scala limitation – it’s just inherent in the different between compile-time and runtime.)
I agree with @tgodzik that macros might work for your needs, but that’s because it’s a compile-time operation. If stuff really can’t be defined until runtime, then it’s not really possible to use it in a typesafe way like that.
1 Like
Good point – I don’t think I’ve seen that used in the wild yet, but it’s an intriguing new technique.
Based on the docs, though, I think it requires knowing and stating the types upfront, even if the implementation is generated at runtime – that’s how it gets around my “impossible” statement above. (Indeed, it’s an interesting illustration of the difference between compile-time types and runtime classes.)
And it isn’t obvious to me whether this can work when starting out with a String at compile time. (Might be possible; I don’t know the new metaprogramming systems deeply enough yet.)
2 Likes
Oh! I missed that. No, pretty sure you cannot start out with a string and use that. Gotta spin up a full compiler for that.
As i understand, the meta programming of scala 3 can be narrowed to only what’s required for the dynamic behavior specified. Not really what happens as far as i know.
@hmcbride
Correct class loader is tb.mirror.classLoader, not compiled.getClass.getClassLoader.
Correct way to work with toolbox-generated classes is with tb.define:
using quasiquotes:
import scala.reflect.runtime.universe.{ClassDef, Quasiquote}
val addressSymbol = tb.define(
q"""
case class Address(
city: Option[String],
country: Option[String],
street: Option[String]
)
""".asInstanceOf[ClassDef]
).asClass
val personSymbol = tb.define(
q"""
case class Person(
address: Option[$addressSymbol],
age: Option[Long],
name: Option[String]
)
""".asInstanceOf[ClassDef]
).asClass
val rootSymbol = tb.define(
q"""
case class Root(
people: Option[Seq[$personSymbol]]
)
""".asInstanceOf[ClassDef]
).asClass
val addressClass = tb.mirror.runtimeClass(addressSymbol)
val personClass = tb.mirror.runtimeClass(personSymbol)
val rootClass = tb.mirror.runtimeClass(rootSymbol)
or parsing strings:
val addressSymbol = tb.define(tb.parse(
"""case class Address(
| city: Option[String],
| country: Option[String],
| street: Option[String]
| )""".stripMargin
).asInstanceOf[ClassDef])
val personSymbol = tb.define(tb.parse(
s"""case class Person(
| address: Option[${addressSymbol.fullName}],
| age: Option[Long],
| name: Option[String]
| )""".stripMargin
).asInstanceOf[ClassDef])
val rootSymbol = tb.define(tb.parse(
s"""case class Root(
| people: Option[Seq[${personSymbol.fullName}]]
| )""".stripMargin
).asInstanceOf[ClassDef])
val toolboxClassLoader = tb.mirror.classLoader
val addressClass = toolboxClassLoader.loadClass(addressSymbol.fullName)
val personClass = toolboxClassLoader.loadClass(personSymbol.fullName)
val rootClass = toolboxClassLoader.loadClass(rootSymbol.fullName)
The package will be not org.example but something like __wrapper$1$b1d11b2e2a4c40bab5387e1f253f30fb.
1 Like
@hmcbride If you want to compile all classes at once, accessing them will be trickier
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox
import scala.jdk.CollectionConverters._
object DynamicCompilation {
def compileCaseClasses(code: String): Option[ClassLoader] = {
try {
val tb = currentMirror.mkToolBox()
val compiled = tb.compile(tb.parse(code))
val classLoader = tb.mirror.classLoader
println(s"Class loader being compiled is : $classLoader")
Some(classLoader)
} catch {
case e: Throwable =>
println(s"Compilation Error: ${e.getMessage}")
e.printStackTrace()
None
}
}
// javaOptions += "--add-opens=java.base/java.lang=ALL-UNNAMED", fork := true
// java 11
val classLoaderClassesField = classOf[ClassLoader].getDeclaredField("classes")
classLoaderClassesField.setAccessible(true)
// java 17
// val classGetDeclaredFields0Method = classOf[Class[?]].getDeclaredMethod("getDeclaredFields0", classOf[Boolean])
// classGetDeclaredFields0Method.setAccessible(true)
// val classLoaderClassesField = classGetDeclaredFields0Method.invoke(classOf[ClassLoader], false)
// .asInstanceOf[Array[java.lang.reflect.Field]]
// .find(_.getName == "classes").get
// classLoaderClassesField.setAccessible(true)
def getClassByName(classLoader: ClassLoader, className: String): Option[Class[_]] = {
try {
val classes = classLoaderClassesField.get(classLoader)
.asInstanceOf[java.util.Vector[Class[?]]] // java 11
// .asInstanceOf[java.util.ArrayList[Class[?]]] // java 17
.asScala
classes
.find(_.getName.contains(className)).map { clazz =>
// ...$2$ is a companion object, ...$1 is a class
classLoader.loadClass(clazz.getName.replace(className + "$2$", className + "$1"))
}
} catch {
case e: ClassNotFoundException =>
println(s"Class $className not found: ${e.getMessage}")
None
}
}
def main(args: Array[String]): Unit = {
val code = """
{
case class Address(
city: Option[String],
country: Option[String],
street: Option[String]
)
case class Person(
address: Option[Address],
age: Option[Long],
name: Option[String]
)
case class Root(
people: Option[Seq[Person]]
)
}
"""
compileCaseClasses(code) match {
case Some(classLoader) =>
getClassByName(classLoader, "Address") match {
case Some(addressClass) =>
println(s"Successfully compiled Address class: $addressClass")
case None => println("Could not find Address class.")
}
getClassByName(classLoader, "Person") match {
case Some(personClass) =>
println(s"Successfully compiled Person class: $personClass")
case None => println("Could not find Person class.")
}
getClassByName(classLoader, "Root") match {
case Some(rootClass) =>
println(s"Successfully compiled Root class: $rootClass")
case None => println("Could not find Root class.")
}
case None =>
println("Compilation failed.")
}
}
}
//Class loader being compiled is : scala.reflect.internal.util.AbstractFileClassLoader@62df0ff3
//Successfully compiled Address class: class __wrapper$1$40b3bbb674b04663934483404ecee51d.__wrapper$1$40b3bbb674b04663934483404ecee51d$Address$1
//Successfully compiled Person class: class __wrapper$1$40b3bbb674b04663934483404ecee51d.__wrapper$1$40b3bbb674b04663934483404ecee51d$Person$1
//Successfully compiled Root class: class __wrapper$1$40b3bbb674b04663934483404ecee51d.__wrapper$1$40b3bbb674b04663934483404ecee51d$Root$1
1 Like
this program dynamically compiles code for Scala 3
import dotty.tools.dotc.Driver
import java.net.{URL, URLClassLoader}
import java.nio.file.{Files, Path}
import java.io.File
object DynamicCompiler:
private val driver = Driver()
def compile(code: String, className: String, outputDir: Path): Boolean =
val sourceFile = Files.createTempFile(className, ".scala").toFile
sourceFile.deleteOnExit()
Files.writeString(sourceFile.toPath, code)
val args = Array(
sourceFile.getAbsolutePath,
"-d",
outputDir.toAbsolutePath.toString,
"-classpath",
System.getProperty("java.class.path")
)
val result = driver.process(args)
!result.hasErrors
def loadClass(className: String, classDir: Path): Option[Class[?]] =
val loader = URLClassLoader(Array(classDir.toUri.toURL), getClass.getClassLoader)
try Some(loader.loadClass(className))
catch case _: ClassNotFoundException => None
def main(args: Array[String]): Unit =
val outDir = Files.createTempDirectory("scala3-compiled")
val objectName = "DynamicRoot"
val className = "Root"
val code =
s"""
|case class Address(city: Option[String], country: Option[String], street: Option[String])
|case class Person(address: Option[Address], age: Option[Long], name: Option[String])
|case class $className(people: Option[Seq[Person]])
|object $objectName
|""".stripMargin
if compile(code, objectName, outDir) then
println(s"Compilation succeeded. Output in: $outDir")
loadClass(objectName, outDir) match
case Some(objCls) => println(s"Loaded object class: $objCls")
case None => println(s"Failed to load object class: $objectName")
loadClass(className, outDir) match
case Some(rootCls) => println(s"Loaded case class: $rootCls")
case None => println(s"Failed to load case class: $className")
else
println("Compilation failed.")
2 Likes