Runtime reflection in Scala

[sangamon]

@jimka

If I parse this in clojure using (json/read-str (slurp "/tmp/small-example.json")) , then I get not a JSON object containing JSON types which I need to study and understand, rather I get a Map whose keys are either maps, or arrays, or numbers, or strings, or lists etc all the way down.

First I’d think that e.g. JsValue shouldn’t require a huge amount of studying. Second, having a restricted set of types is a feature. With a JsValue, there’s exactly 8 different forms it can take. A Map[String, Any] can literally contain anything. I’ll have to guess which types are used to encode JSON arrays/numbers/…, I can’t rely on the compiler to tell me when I’ve forgotten to cover one of the cases, and so on.

@jimka

I don’t need to be an expert at JSON parsing libraries to find the coordinates and figure out if each is array of array of array of array, or simply array of array of array.

I think you are conflating two things here. The JsValue structure is completely equivalent to the Map[String, Any] structure, and you could do the same check for array nesting depth in both. The convention of adding polymorphic subtype information as JSON attributes just makes this distinction easier, and it covers more ambiguous cases - imagine a list that can contain both absolute and relative coordinate pairs, both consisting of two numbers.

@jimka

What I wanted to do with this was build a map of country name to list of perimeters describing its border. Once that mapping was created, then I was very happy to have well typed functions to manipulate them. But extracting that data in Scala was really difficult, especially for a non-expert.

That’s the part you have completely omitted on the Clojure side. The equivalent of json/read-str (slurp ...) with spray-json would simply be JsonParser("/tmp/small-example.json"). That’s the format-level part, and the resulting structure is fully equivalent. Converting to the domain level (i.e. Map[String,List[Perimeter]]) is the interesting part. (And I think I recall that I had given some suggestions at the time how this could be approached a bit more conveniently.)

And again, if you don’t like JsObject and friends, a conversion to Map[String, Any] should be feasible in a single, recursive pattern-matching function, if I’m not missing anything.

[curoli]

I would not characterize the philosophy of dynamically typed languages as “data is unsafe/hostile and needs to be checked all the time”, but rather as “it’s up to the user to make sure the data has the correct type”

In many dynamically typed languages, providing an argument of the wrong type will rarely result in a type error and more likely result in unexpected behavior. JavaScript is particularly notorious for that.

In a statically typed language, if you have input of unknown type, the first thing is usually to project it to some expected type and flag it as an error if it is not. Therefore, Scala libraries that deal with querying a database or parsing JSON allow you to define precisely what type you expect and will deliver you an object of that type or an error.

[jducoeur]

(Context: I’ve been a full-time Scala teacher, and still spend a fair bit of my time tutoring new folks in the language, including completely new programmers as part of ScalaBridge. I get the “put yourself in the student’s shoes” thing.)

@jimka

In the case of the Scala code above, it is really a lot of code involving advanced concepts (cursors, walking up and down, implicit decoder, lots of new type names) to do something which is simple. I recall spending several days on it, because it was failing to parse the file and giving me no usable feedback about what I was doing wrong.

Right – my point is, that’s all optional. You need that in order to get the strongest possible types from Circe. You’re saying that you don’t want strong types, which is fine – don’t use them. Instead, use a JSON parser designed to support a less strongly-typed version, like play-json, uPickle or weePickle.

I mean, the total parse code for play-json is:

Json.parse(inputString)

That’s it – and that gives you an AST that is structurally identical to the one you say you’re looking for. Yes, you have to explain that JsObject is a specialized version of Map that specifically means that this is a JSON Map, but that’s not a hard concept, and it helps get across the Scala way of thinking, that Types Are Good.

(The actual functions and types are different for uPickle and weePickle, but otherwise they’re identical to this.)

All the effort is in producing strong types; those are much easier to work with in nearly every business case, but they are entirely optional. If what you want is a simple AST, just use a simple AST.

[jducoeur]

I should note an important corollary: you’re thinking about this as “this is how Scala does it”. That’s usually wrong – it’s all about how a specific library does it. If you don’t like the way that library works, there tends to be another one that is more like what you want…

[sangamon]

@jimka

What I wanted to do with this was build a map of country name to list of perimeters describing its border. Once that mapping was created, then I was very happy to have well typed functions to manipulate them. But extracting that data in Scala was really difficult, especially for a non-expert.

I’ve just revisited that thread and your code, and I still cannot really understand your complaints. I still like my spray-json/lenses approach a bit better, and I probably would do a few things different than you with circe, but I don’t think that the HCursor API is that terrible. I’m really wondering how an Any-based implementation should fare significantly better.

In case you’re curious to try it out, here’s a naive conversion from circe Json to Map[String, Any]:

def toUntyped(json: Json): Option[Map[String, Any]] = {
  def cnv(j: Json): Any =
    j.fold(
      null,
      identity,
      _.toDouble,
      identity,
      _.toList.map(cnv),
      cnvObj
    )
  def cnvObj(jo: JsonObject): Map[String, Any] =
    jo.toMap.view.mapValues(cnv).toMap
  json.asObject.map(cnvObj)
}

Another thing is that your use case is somewhat special, though not completely unusual. You are reading the JSON data while transforming/extracting at the same time. This is perfectly fine, but the more common case is that one really wants to map JSON data to an equivalent representation in the code. If we take this route, the picture is somewhat different.

import io.circe.generic.auto._

sealed trait Geometry
case class Polygon(coordinates: List[List[List[Double]]]) extends Geometry
case class MultiPolygon(coordinates: List[List[List[List[Double]]]]) extends Geometry
case class Properties(name: String)
case class Feature(properties: Properties, geometry: Geometry)
case class FeatureCollection(features: List[Feature])

implicit val decodeGeometry: Decoder[Geometry] =
  (c: HCursor) =>
    c.downField("type").as[String].flatMap {
      case "Polygon" => c.as[Polygon]
      case "MultiPolygon" => c.as[MultiPolygon]
    }

val features = parse(jsonStr).flatMap(_.as[FeatureCollection])

circe in particular is pretty much opiniated in that regard, as expressed in the design document:

You generally shouldn’t need or want to work with JSON ASTs directly.

Once you have this direct representation of the JSON model, it should be much easier to transform it into the representation your application actually wants, i.e. Map[String, List[List[Location]]]. This is certainly less efficient than the direct transformation upon parsing - on the other hand now you have the full content of the JSON data available in a nicely typed fashion for whatever else you may want to repurpose it to.

[jimka]

@sangamon

scala> 123.getClass.getClass val res0: Class[Int] = int scala> 123.getClass.getClass val res1: Class[_ <: Class[Int]] = class java.lang.Class scala> Vector(1, 2, 3).getClass val res2: Class[_ <: scala.collection.immutable.Vector[Int]] = class scala.collection.immutable.Vector1 scala> Vector(1, 2, 3).getClass.getClass ^ warning: inferred existential type Class[T] forSome { type T <: Class[T]; type T <: scala.collection.immutable.Vector[Int] }, which cannot be expressed by wildcards, should be enabled by making the implicit value scala.language.existentials visible. This can be achieved by adding the import clause ‘import scala.language.existentials’ or by setting the compiler option -language:existentials. See the Scaladoc for value scala.language.existentials for a discussion why the feature should be explicitly enabled. val

HI Sangamon, is what you copied into that message correct? Maybe I’m blind, but I don’t see the difference between the first two expressions which evaluate differently.
123.getClass.getClass vs 123.getClass.getClass

[SethTisue]

@jimka

123.getClass

The first 123.getClass.getClass should have just been 123.getClass

[sangamon]

Yeah, sorry, Murphy’s law - I think I missed the first line when copying the REPL session and then somehow managed to grab the wrong line when trying to fix it up. I’ll edit it for posteriority.

[sangamon]

@jimka

In a dynamically typed language, a function may accept unsafe input data and validate it before proceeding.

Not really intending to resurrect this thread, but I just came across a blog post that reminded me of this discussion: Parse, don’t validate.