Making scala easier and more popular, with compiler level switches

Question
21

It’s true that in terms of the ecosystem things can be pretty iffy, but I think that again, it’s a lot about documentation so that you aren’t left fending for yourself.

Specifically, for someone not interested in advanced fp, for databases I would recommend scalikejcbc, quill, or jooq (a Java library with a good API). For an http client, https://github.com/lihaoyi/requests-scala is very simple to use.

IMO http4s is for people that are sold on the necessity of pure FP and therefore are willing to put in effort. Similarly, don’t use Slick unless you’re sold on what it buys you.

Now that’s easy for me to say, but how are most beginners to know that?

Which is why I say, exactly. The problem is that there isn’t documentation guiding beginners about all of this, that beginners will come across. Thus for people that are comfortable scanning lots of things and trying things out, that thrill in learning interesting things, they can find they way, but everyone else won’t bother trying.

22

Calling it a PR issue is interesting, because I’ve been observing that we communicate most easily with those with whom we are similar in our way of thinking: so the scala creators’ communication has resonated more with those who are deeply interested in language design, compared to most who use python or java, etc.

The proposal makes it so you can ignore features in a predictable, organized way, instead of a chaotic or ad-hoc way. The “chaotic” way (where you use what you want, hope somebody else can understand it, and good luck if not) is not persuasive when arguing for use of a language, in my corporate experience. (Maybe we are agreeing on that part.)

With the proposal, the value of the language could be there easily (I hope) both for you, and for those programmers who are less interested in all the things scala can do, and who just want to code.

Again my point there are real potential benefits to many devs from this proposal, with low cost (I hope): zero cost (besides one project config setting) for those who like it the way it is, and large benefits (listed earlier) for those who want to use scala on more projects and with more peer types.

And I think that would dramatically allow changing the perception problem that you mentioned (especially if the levels and names are chosen carefully – new language: “Filbert” or “Passo” or such = scala level 1, or both 1 and 2 – and instructional materials are changed to match). So, even if you think it doesn’t help you, doesn’t it seem like it could help with adoption by others, and in working as well-managed teams with scala devs at different levels, given the bullet-list of benefits I made earlier?

(Allowing progress by levels also fits with the quote on the name “Scala”, from an old FAQ: “First, ‘scala’ is the Italian word for stairway, which is appropriate since Scala helps you ascend to a better programming language.” Yay!)

23

I personally do not feel that there is anything wrong or complex about Scala that any java developer could not grasp.
However, there are two very, very important Obstacles to learning Scala:

  1. POOR DOCUMENTATION.
    The fact the the books and on-line documentation for scala are written in an EXTREMELY user unfriendly manner turns most other-language developers off immediately.
    Here is an example:
    Open a book on , say , AKKA actors. The book is usually written in a way that requires you to have a full grasp of all types of aspects of the Scala Eco-system. To most Java developers it will eventually dissolve into babble or greek. Usually such books will expect you to already know extremely complex things about the language. These books usually do not contain workable complete code examples and they are almost always DEPRECATED. And most of the books and documentation take an almost immediate deep dive into esoteric functional programming.

  2. Absolutely HORRIBLE build eco system.
    This is very controversial among the hard-core Scala Developers. They both enjoy and embrace SBT and other non-mainstream build tools. But to developers in most enterprise environments, this is a big head-ache and hassle.

Now I am a java developer that is attempting to learn and use Scala on my job. I am slowly acquiring expertise in the language but IT IS OVERWHELMINGLY difficult to do this. I sometimes wonder about the mindset of the authors of various scala books. How do they view potential audiences? Do they think that everyone is working towards upper academic degrees in functional programming theory and research? Do they feel that the average developer is interested in creating new languages or new idioms in distributed processing ?
Hopefully these attitudes will change before they destroy any hope of the Scala every gaining wide acceptance.

1 Like
24

I have to reply to this because your first paragraph is exactly what I bring up when I describe teaching Scala to novice programmers. I find that I can do introductory programming rather well in Scala without hitting a lot of the dark corners early on. Now the rest of this post deals with libraries, and that is where some of this breaks down for Scala. Libraries often involve some of the more complex features of the language. So introducing experienced programmers to Scala could actually be a lot harder than introducing novice programmers to Scala because the experienced programmers have a higher expectation bar for what their early programs should be like.

25

If defined levels existed for the compiler like I’ve described, they hopefully would also be used in discussions on libraries and as a guide to orient documentation.
In other words: as an organizer for all scala materials, with clear stepping stones to the next level, and what is or could be involved, for a team or individual to make a clear decision. From your comments and my experiences also, it really sounds like it would make life easier for many.

26

Here are some ideas (just brainstorming) for equivalent ways to call the compiler at each level (I think “passo” is Italian for “step”):

scalac -L1
passo1

scalac -L2
passo2

scalac -L3
passo3

scalac (no added switch so fully backward compatible)
passo4

Then instructional materials/libraries could be labeled as to if they were recommended for “Scala LN” or “passoN”, or for some who prefer, there is the old: “all of it at once” approach. Sounds like that would make some people happy. And the “passoN” numbering would motivate some learners purely for the challenge, and because they know clearly what the next step involves.

27

lol you just explained what is wrong with scala today. In theory, programming languages are supposed to be easy to use. As they evolve, they theoretically should become easier and easier to use and to do more and more complex things. That is not the case with Scala. Scala requires a deep understanding of Functional programming in the Academic sense … at least from a java-centric point of view , that seems to be the case to me.

28

Interesting discussion. I googled http request in scala and found this:
https://stackoverflow.com/questions/11719373/doing-http-request-in-scala

val result = Http("[http://example.com/url](http://example.com/url)").postData("""{"id":"12","json":"data"}""")
  .header("Content-Type", "application/json")
  .header("Charset", "UTF-8")
  .option(HttpOptions.readTimeout(10000)).asString

Seems easy and simple enough :slight_smile:

IMHO, the problem is not language complexity or lack of simple libraries (you can always use Java libraries too). I find that many Scala programmers like to show off by unnecessarily using advanced features. Some take pride in the fact that others can’t read their code! There is also a cult of using “idiomatic scala” which I call “idiotic scala”. The language is awesome and I love it but the community is kinda exclusive and takes pride in it.

2 Likes
29

However, that code is not usable in an enterprise environment. For example, say you have a Spring Boot Micro-service containing various rest services. And you want to use scala to easily use the services. You would need to send HTTP rest client calls to the service and easily convert the request and response to Scala Objects. There are various ways to do this in scala … but … the techniques are not well documented. One will have to study, look between the lines, try different things, scan google for viable examples … blah blah and blah. Therein lies a major problem with Scala. Only extreme programmers have the time or patience to do that. Most Senior level java developers do not have the time for such study.
This is not to criticize your code example. I am just trying to describe what most senior level java developers would need to do.
Let me describe what I, should I try to use your http code above, need to do.:
The code would have to be multi-threaded since I would most likely be sending thousands of such requests. The method would have to be generic since there would be different request and response types. Cleaning up would have to be automated in some way, i.e. closing the http connection and/or setting up some type of connection pool that is transparent to the developer.

This is how senior level java developers develop. So when they decide to learn Scala, they still want to develop in that manner. Which means that they need to understand generic programming, multi-threading … etc etc … in the Scala Eco System. It is difficult for many developers to do that in Scala because the documentation does not easily help them to systematically gain that type of expertise in Scala.
AND … many just give up.

30

One more thing is this:
Writing complicated code that no one understands never works in the financial world except in RARE cases. Lots of consultants come and go. It is a financial burden to have to maintain code that no one but a few can understand. For example, If I took one of those examples from “Functional Programming In Scala” and showed it to the typical java developer they would throw their hands up and call it Greek.

1 Like
31

Writing complicated code that no one understands never works anywhere.

Pretending that if you don’t understand it, it must be complicated code that nobody understand might work or not work, depending on who you’re talking to.

32

Thanks much, for all the illustrative comments. At the risk of being repetitive, some summarizing, and a question:

I think we’ve shown the idea that scala would be more useful for many, including for more types of projects with more types of peers, if it had compiler-enforced (and maybe community-enforced) complexity levels, which somehow accounted for instruction and library usability or recommendations (where practical) to match an explicitly indicated difficulty level, and that each project may choose its level explicitly (or doing nothing stays is at the highest level, the same as today). What exactly would be in those levels would be an interesting but separate discussion that I hope could be managed without excess trouble.

Some general benefits are listed in my earlier bulleted list, with the suggestion that the first level just lets average java programmers do what they are now doing with slightly different syntax and practically no new learning effort, but room to grow in manageable, predictable steps.

I would like so much to be able to recommend using scala (or ~ “passo1”) in more situations (including the niche kotlin is trying to fill), and this would help with that, and would remove a barrier to growth to other levels of programming, among some programmers who otherwise would stay at a basic level only or have to do much more work to get the growth.

Question: Would there be harm in posting a message referring to this discussion, seeking comments on how hard a very basic implementation of this would be at the compiler level now, and if or how it could integrate with the dotty vision & timeframe, in the “scala contributors” list?

33

Posting on contributors asking about the general viability sounds fine – though we already have this in language imports. The technical issue is unlikely to be the problem

More of a problem is probably the subjectiveness of what is a more advanced feature, and what is a less advanced feature. If you asked several different people to order a list of language features of being advanced or basic, you would probably get wildly different orderings back.

For example, I would suggest a beginner never to use null, var, return or asInstanceOf. All of those language features have very useful uses, but they’re IMO very advanced uses that make code difficult to write, read, debug, and fully understand. I would never suggest these things to be enabled at a beginner level.

If the user is already familiar with these features from e.g. Java, and they want to learn how to “write Java in scala”, then they’d need those features, but don’t need a bunch of others, that I’d consider core to scala (e.g. function values, sealed trait hierarchies, implicit parameters, for comprehensions).

That makes IMO a language level approach that caters to one segment of new scala learners counter to all others.

1 Like
34

Thanks for making those points. Certainly those things (avoiding null, etc) are part of “scala goodness”, and maybe you are right to put them in level 1, or maybe they go in level 2. I think it depends on whether the value of allowing simple on-boarding is greater or less than the value of those features in level 1. Of course nobody is required to use the restricted levels, but only those teams or projects that find it is to their advantage.

I suggest something like this to avoid huge endless debate (but others certainly might have better ideas):

  1. Make a list of language features.

  2. Rank the list by “core-ness” or frequency of use, by taking a poll and/or scanning available code). Limit debate (but allow market testing in later steps if time-bound and someone wants to pay for it).

  3. Level one contains the features found in java (as used by the average programmer; jdk5? other?), plus maybe a very few others – the goal is what can be learned from a web page or “equivalency cheat sheet” in an hour or less – and the other key language differences like “public scope is default” and “the last line’s value is returned” should be included in the hour (or 2). This level 1 feature scope and learning time limit should be tested on some devs who are new to scala and not excited about it, for accuracy and feedback – because those are the ones who would try to veto its use in a project or organization if they don’t like it.

  4. Split the rest of the prioritized feature list into 3 parts (maybe grouping some conceptually related things together if near a cutoff and they are at the same level of difficulty as each other, then cut it somewhat arbitrarily–as an initial experiment?), which become levels 2-4. We might default to restrictiveness when in doubt, because it might be easier to move features down in levels than up, to avoid breaking code if we have to move features around in the future: ie, move features down but not up, for future backward compatibility, but consider the cost to training & migration of java developers etc. Or maybe we would never have to move features later because a project can just change levels or maybe add extraFeatures in the config (or parameters).

  5. change the compiler:
    allowedCode = levelDefinedFeatures + extraFeatures
    if (someCode not in allowedCode) fail with “Language feature xyz not in passoX/scala level X, nor in extraFeatures specified to compiler”.

  6. if someone feels restricted, they change levels, add a specifiedException, or just use the default compiler which has no restrictions.

  7. Try it. Wait. Apply lessons learned for a future major release (or in dotty).

I hope that is more do-able than requiring perfection or universal agreement before moving forward. And again, I think it could help a lot as outlined and discussed earlier by answering some criticisms and enabling adopters and learners at any level, improving collaboration, and reducing the cost of using scala at some level.

So, any effort would start at the level that makes the most sense for their goals and background. Anyone can do what they want, while it aids communication in many ways and for certain kinds of teams as discussed.

35

Yes, maybe the level definitions can be done at the granularity of imports, I don’t have a feel for how well that works to define helpful cognitive boundaries. Maybe it is an easier way to scan code and decide what goes where, and managing what libraries go where (based on their imports–or not). I don’t know because some of the hard-to-read-for-newbies things are built in, not imported.

But either way, it should be a compiler switch (and/or alias) for simplicity and convenience of adoption.

36

This sounds like a massive amount of work, and will result in an oddly fragmented and unusable ecosystem for a long time, as most things won’t initially fit neatly into any of the simplified variants that are defined, pretty much whatever they are.

I have a simpler idea for level switches for people coming from Java.

Level 0: Java. Training time: zero, you already know it. (Or maybe not, if a new Java version is out, but the initial learning time is pretty quick.)

Level 1: Scala. Training time: a while; be prepared to focus hard on learning new things that will make you a better programmer.

If you have a lot of programmers at level 0, but some use level 1, then it’s the responsibility of the level 1 folks to write wrapper APIs so their stuff can be consumed by those at level 0.

There is plenty of precedent for this. Akka, Play, Spark, etc., all have Java APIs and Scala APIs. It does make things a bit harder, but mostly the people implementing the libraries/frameworks would rather write a Java API with most of the implementation in Scala than to use a subset of Java that was functionally indistinguishable from Scala save for some superficial syntax that can be learned in a few hours.

Now, you do have to decide, as an organization, what style of Scala you’re going to be using. The FP-purist approach has some advantages, but you really only get those advantages once you learn a new way of doing a huge number of common tasks. It’s not impossible; it’s considerable work compared to training someone who knows Java to use Scala in a more Java-like way because you don’t have to teach them Java. A lot of knowledge is already helpful. With the FP-purist approach, if you’re starting in Java, much less of your existing knowledge is relevant. (The reverse is true for people coming from Haskell who don’t know Java or other eager, side-effect-friendly languages; there just aren’t nearly so many such people.)

But you can’t tell the compiler, “We want to use Java-like Scala, not FP-purist Scala” because those are two different ways to use the same features. Java-like Scala is a great language precisely because it has those features. Those features are so powerful and general that you can almost emulate a Haskell-like style of programming using them. Are you using implicits to provide context for your actor executors? Or are you using them to define a monad typeclass? Are you using higher-kinded types to define StateT, or to specify the proper type relationships between your mutable collections?

Even though the compiler can’t easily tell the differences, programmers can. So the idiomatic style for the company should be maintained by code review and training. You want it that way anyway because you don’t want your style fragmented every time a new person is hired.

So my recommendation is to not learn Scala until you want to learn quite a lot of it. It’s not so hard. There are some really great books out there. (The free online stuff could be better, admittedly.) But you need to devote some time, and some thought, and if you know Java don’t think it’s just a quick upgrade. There are deep and powerful capabilities that you can get familiar enough to know what you’re looking at in a few days, especially with some good books and/or good training, but which take a while to master (just like Java generics take a while to master).

4 Likes
37

Having multiple languages for the levels or types of projects & programmers makes it more costly overall, because you have to remember and maintain standards etc for multiple languages to work with them all, and it is harder to stay current with best practices, tooling, etc in more languages over time, than with fewer. And the big difference (in everything) when going to the next level discourages doing it, for many.

I don’t see how it is a massive amount of work: list features, sort them (somehow: brute force), divide into sets, and implement (but again, I’m not a compiler maintainer). Those who want to ignore it completely, can. I had the idea to do it with training and code reviews, and tried a little, but that is much harder to do in an ongoing/consistent way, especially if one does not rule the environment, because the lines are not clear, and it is easier to slip and bend the rules until they are gone and useless–and someone starts saying “scala is too hard” so you can’t any longer work with multiple kinds of projects/programmers all in scala. It is too hard to define and enforce, and ends up not happening (some experience). Therefore too hard to recommend scala to “regular” programmers one is likely to work with or recruit, or to use it on those projects.

And anyone can ignore it and just compile as they did before.

38

sort them (somehow: brute force)

What do you mean by this?

39

There are a lot of problems with this one.

This could make a neat set of features for programmers who are very experienced in Java 5, and not experienced in anything else. But if you’re limiting yourself to this, there is very little use in using scala as opposed to, say, modern Java or Kotlin.

That sounds as an odd niche to cater to with general purpose language levels. Why do you want to focus on these newcomers in specific with those language levels? Why not people with no programming experience at all, or only Python, or only Haskell, or only JavaScript, or only C++, or only Go?

40

Aside from the philosophical aspect of this discussion, I don’t even see how this would technically be possible/feasible. If you have a basic version of Scala where case classes don’t exist, implicits don’t exist, even higher order functions apparently don’t exist, how would a user of that basic Scala even use the standard library which intensively uses all of those features. You would need to have separate standard libraries for every Scala level. Then we’re not even talking about third party libraries. It’s an inherent property of library writers that they have a very decent understanding of the language, so they are very unlikely to write libraries in level 1 Scala. Hence there are almost no Scala libraries that starting Scala users can interface with. Not even Play, Akka, Spark …