Using Slick 3.2.0 provides classic example of why 'implicit' drives people out of Scala


#1

Although I cannot use the Scala language at work because the productivity of our team would crater, I have still an interest in a language I would love to be able to use in a business world. Deciding that perhaps I was too harsh on the language, I have been playing with a couple of Scala libraries to try and give the language another shot in my off hours work. I just spent the better part of 7 hours playing with Slick 3.2.0 in Akka and Scala. I was massively frustrated and it turned out that yet again it is an implicit conversion that hosed me. Consider the following slick project.

Users.scala

package persistence.entities

import slick.driver.H2Driver.api._

case class User(id: Long, name: String)

class UserTable(tag: Tag) extends Table[User](tag, "user") {
  def id = column[Long]("id", O.PrimaryKey, O.AutoInc)

  def name = column[String]("name")

  def * = (id, name) <> (User.tupled, User.unapply)
}

UsersDAL.scala

package persistence.dal

import persistence.entities.{Tables, User, UserTable}
import slick.jdbc.JdbcProfile
import slick.lifted.Query

import scala.concurrent.Future

trait UserDAL {
  def findById(id: Long) : Future[Option[User]]
}


class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL {
  override def findById(userId: Long)  = null

  def foo (userId : Long) = {
    val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId)
    db.run(filter.result)
  }
}

Should be simple right? No. The db.run(filter.result) line does not compile on the result call. OK, time to comb the Slick documentation. No dice, my code looks just like theirs does fundamentally. So why is the result method not available. Do I have the wrong type? Nope. What about the wrong library? Nope. Update from 3.1.1 to 3.2.0 ? Done … but … no. What about the wrong structure for my table? Am I missing something? Comb example code … and nope. OK, I am frustrated as all get out. In the meantime my irritation grows when I see example code like the following.

 override def insert(rows: Seq[A]): Future[Seq[Long]] = {
    db.run(tableQ returning tableQ.map(_.id) ++= rows.filter(_.isValid))
 }

Good god, how to decode that mess. Obviously run is taking something as a parameter but exactly what is the order of operations here? Anyway, I digress.

Back to the subject at hand, I am at a complete loss for how to explain the problem here so, I decide to create a post on the Slick forums to figure out what I am doing wrong. I think, for simplicity sake lets combine all of our classes into one file.

package persistence.entities

import java.time.Instant
import slick.driver.H2Driver.api._
import slick.jdbc.JdbcProfile
import slick.lifted.{Query, TableQuery}
import scala.concurrent.Future

case class User(id: Long, name: String)

class UserTable(tag: Tag) extends Table[User](tag, "user") {
  def id = column[Long]("id", O.PrimaryKey, O.AutoInc)

  def name = column[String]("name")

  def * = (id, name) <> (User.tupled, User.unapply)
}

object Users {
  val users = TableQuery[UserTable]

  trait UserDAL {
    def findById(id: Long) : Future[Option[User]]
  }

  class UserDALImpl(implicit val db: JdbcProfile#Backend#Database) extends UserDAL {
    override def findById(userId: Long)  = null

    def foo (userId : Long) = {
      val filter: Query[UserTable, User, Seq] = Tables.users.filter(_.id == userId)
      db.run(filter.result)
    }
  }
}

OK, now that we have combined them we are ready to post this … but wait … it compiles now. What The Fork? Oh wait. There was an implicit conversion going on in there somewhere. There was something in the new file converting our query type into something that had the result method predefined. Magically, behind the scenes and with the right import it doesn’t happen and the code doesn’t compile.After experimentally deleting imports i discover it’s import slick.driver.H2Driver.api._ that is the home of the implicit conversion. So I go look at the file and nope, no implicit. I drill and drill and finally find it. Wonderful. 7+ hours wasted for this. The fun part is the implicit conversion happened as a result of the implicit conversion of an implicit conversion of an implicit conversion. Do the designers of Scala think this is actually good? You are just supposed to “know” that the conversion of a conversion of a conversion is happening I guess. Or is the goal to make the code so cryptic and opaque that the source of the library is entirely unreadable?

Given my originalUsersDAL file there was no way for me to know that to make the code compile I had to find some implicit conversion that could turn a Query into a DBIOAction. There is no means for you to know where this conversion is, what file has to be imported. All of the example code just assumes you must already know. Its fine copying the example code but if you modify, reorganize code into many files you break the delicately balanced Jenga tower and it doesn’t work.

So after playing around for a few days with this project, I can see that if I ever brought this code into a business environment my CEO would absolutely murder me as it would take 6 months to add the most basic of features, not to mention 1+ years to train a new developer to be even moderately productive. Once again, its the implicit and its use in the language that is at the core of the problem. It actually makes me sad, but it would be wholesale irresponsible of me as a Principal Software Architect to recommend these techs in a business that makes its money putting out product for users in a short time.

So why post this? Because otherwise i LIKE Scala and wish it could be different. But its becoming like the JDK, they have decided that is the direction they are going and no one on the planet can convince them otherwise. OK now that I am done stating my opinion, I am ready to get flamed again. :slight_smile:

– Robert Simmons Jr.


#2

Out of curiosity, what was the compiler error you were getting?


#3

Gotta say Slick is the WORST example of this. The documentation is terrible at calling out which imports you need to provide the implicits. I’ve been using Scala for years, and I still can’t get a new Scala app running in less than about 4 hours with googling and googling and googling. It’s a failure in documentation as much as anything here. Really frustration.


#4

The problem with Slick isn’t isn’t implicits, it’s path dependent types, and it’s caused by Slick’s bad design, not by Scala.

Importing the right things, from inside values (not packages), is a problem I suffer from with Slick, and it’s the same whether the imports are implicit or explicit.

To make this worse, you need to import from a specific driver to define your class extending Table. At work I need to write code that supports multiple databases! I had to write a complex looking thing that imports the type of my parent class from the right driver based on runtime configuration, and make all my methods operate on the right path-dependent types. And then I had to write tests to instantiate the different class definitions of each driver we support. This is not a sane approach to writing DB-agnostic code.

The Slick query DSL is also inconvenient. I hope I have the time to eventually move the codebase to Quill or another alternative.


#5

I think it’s unreasonable to expect Slick’s API to satisfy everyone’s design expectations, and it seems weird to me that people get so angry at it, as if they have no choice in the matter. There are a bunch of perfectly reasonable DB layers to choose from. Every design has tradeoffs, and Slick works very well for lots of people.


#6

tpolecat: No one (that is a reasonable person) expects slick to work perfectly for everyone. People get so angry at Slick because it looks very popular, reasonable, and straightforward initially, and actually is pretty easy to get started with. The promises of its typesafe DSL are incredibly seductive and alluring.

Before long, though, the (incredible, severe) downsides start showing up, and they just get worse and worse over time.

  • New developers inevitably pay the multi-hour "forgot to import the driver.api._" tax.
  • Compile times start being measured in multi-digit minutes
  • Type errors are unintelligible
  • Implicit resolution failures are unintelligible
  • It effectively breaks an IDE
  • The project is dead / effectively unmaintained
  • There are probably 5-10 people in the world who thoroughly understand slick’s internals (and they are far too busy with other projects to help everyone), and there do not appear to be documentation or even high-level explanation about the internal design, organization, or philosophy available on the internet! So when something breaks its internals, it becomes a show-stopper with effectively unbounded productivity loss.

My team uses Slick. Had i known even 2 or 3 of the above, i’d have avoided it, and the first chance we have the realistic opportunity to migrate away, we probably will. Alas, none of those issues were particularly obvious to me until i had a bunch more experience with it, and had established precedent of using it often in our codebase.

Don’t get me wrong, I’m constantly amazed at the cleverness of some parts of slick’s internals. But the nonobvious negatives far outweigh the benefits in my opinion, and those negatives are not obvious or made clear in a way that can be reasonably considered before heading well down the rabbit hole.

It’s a problem that many popular scala libraries share though, so i suppose slick is in good company here.


#7

How is slick dead/effectively unmaintained? That’s not what I’m seeing.

Also I’m using slick in production and haven’t had a problem with it in 2 years


#8

@Daxten

  • 1 release in the last 15 months, containing basically just a bugfix for a longstanding critical bug.
  • Looking over the commit log in github makes it clear there’s not really been serious development effort
  • The project is “community-maintained” apparently, but there’s not much community development (possibly because no one really understands the internals beyond the primary authors?)
  • Critical issues on the tracker are generally left open for years
  • Even the gitter slick room has a remarkably low percentage of questions or issues actually answered or addressed.

Overall the level of engagement seems incredibly low to an outside observer who actually takes a pretty active effort to be involved.

I could certainly be wrong (and hope i am), but all evidence i see points the other way.


#9

My main issue with Slick was dealing with tables with more than 22 columns, which forced me to dig into its use of shapeless and was pretty rough… I got it working after a lot of sweat, but swore never to use it again!


#10

Wait, it doesn’t actually use shapeless, right? IIRC slick has its own HList implementation that it uses (although that’s as of 3.1 - 3.2 era, can’t speak about prior)


#11

it’s possible actually, i did that a while ago and mostly remember dealing with the HList construct, which may very well have been their own!


#12

I’m not sure the Slick documentation says so, but there might be high-level explanations (at least for the frontend DSL of the lifted embedding) in the documentation for Lightweight Modular Staging. That explains the ideas behind getting an AST from the Scala code you write.

Transforming that AST to SQL is another question, but that is a bit more “standard”—because people have been doing it, in better or worse ways, since LINQ.

One important property (which usually failed in LINQ) is avalanche safety: a query avalanche is when there’s a loop on data outside the DB generating more subqueries, so that the total number of queries depends on the data, and the database cannot optimize across those queries. An avalanche is bad for performance. Avalanche safety prevents that—see papers [Avalanche-Safe LINQ Compilation]
(http://vldb.org/pvldb/vldb2010/papers/R14.pdf) and A Practical Theory of Language-Integrated Query.


#13

I have spent more than a year on a Slick project (a classic database management system). I was pushed to use it by the Play framework, which according to the docs supports basically nothing else than Slick and classic JDBC. It works, but it was one of the worst programming experience I ever had, and indeed I started to find Scala disgusting because of Slick’s issues, which is unfair.

After several weeks of work lost battling with implicits (like you can insert but not delete, sic), saved by miraculous StackOverflow posts from the rare people that understand Slick better than what the documentation allows, I end up with a code that is hard to maintain, just because I am afraid to try any change that will cost me even more time.

Implicits. HLists. Erk.


#14

Here is a link where I started something very similar in the slick group about 2 years ago:
https://groups.google.com/forum/#!search/My$20vague$20and$20honest$20opinion$20about$20SLICK|sort:date/scalaquery/OK_QgYSWA9M/REYRPXBgp54J

I found some of the slick devs understood the issue. However, some other devs were completely dissociated from the problem. I think there is a psychological disconnect between many slick users and slick developers.


#15

Agreed. Get out of Slick whenever you can, as often as you can, for as long you can. Ignore that whacky teammate who just wants to bad mouth it yet keep fixing it to show how smart he/she is.


#16

Interesting, I thought the SQL framework recommended by Play is Anorm
https://www.playframework.com/documentation/2.6.x/ScalaAnorm.


#17

As far as I understand Play is pretty database framework agnostic. There’s the PlaySlick integration library but apart from that chapter in the documentation I don’t think Slick is being pushed anywhere.


#18

Play doesn’t care about the persistence layer. Play is one of the most
modular web frameworks. At the same time it is designed to be very easy to
get started with, so the documentation tends to focus on certain setups
(e.g. routes file and guice DI).

One of the reasons Slick’s codebase is so complex is that it was originally
designed to have broader uses, so they have a fine-grained cake, with
JDBC-specific code on top of SQL-specific code on top of
relational-specific code on top of more general code. So the bottom layer
could be reused for e.g. a document-oriented database, and a lot of the
code could be reused for a non-JDBC driver. At least in principle… I
don’t know to what extent the code follows the architecture.

Also it has support for more advanced use cases, like surfacing reads vs.
writes in the type system and support for Reactive Streams.

That said, like any project without a maintainer, the execution is lagging
far behind the vision. It’s too bad Lightbend had to take Stefan off it.
Maybe now that the collections redesign is wrapping up there’s hope of him
returning to it? In any case they really need to put someone on it. It
doesn’t put them in a good light, Slick was a major project of theirs.


#19

Don’t get fooled like I was. Don’t even use Play, by the way, but Slick
is the worst programming experience I ever had, and I started to find Scala
disgusting. Unfair? Well, then why does the language even allow to do this?

If you want a programming language that limits your expressiveness to
things that are determined safe by the powers that be, you might prefer
Java or Go.

Personally I enjoy speaking English because it lets you express anything
you want… even if that means it allows people to say things that are
incoherent or hard to understand. Similarly I enjoy using a programming
language that lets me express the unique nature of whatever problem I
happen to be solving, rather than boxing me into a limited set of tools
that might make a jobs they weren’t designed for more tedious than
necessary.


#20

If it was planned to be so powerful why was it an authentic nightmare to support something as mundane as an autoincrement indexed key?
I can’t remember how many hours of research and pointless stackoverflow discussions and bad documentation I had to go through to find how to work with a model that contains an autoincrement index.