I’ve become a fan of the less mainstream in-source unit testing style from languages like Rust (where tests are defined underneath function definitions in the same file, rather than in separate test/ or .test locations. Serves as self-documentation, easier to find, “locality of behavior” etc.).
No unit testing frameworks in Scala/Java support this, but seeing as it’s finding success and love from users in other languages, and there are compelling arguments that it is the superior style, it would be nice to have as an option in Scala.
I am wondering if it’s feasible to implement given Scala’s current semantics and infrastructure. My immediate thought is to create a @test decorator to put above testing functions, i.e
Do you want to ship your tests with the production code in published artifacts / executables, then?
You could try using JUnit5’s @Test annotation and put your test methods in the same class or object that houses your SUT methods. I’m assuming that all your shipped code ultimately lives in one or the other. Unfortunately you can’t make the test methods private, so they form part of the API, but perhaps that’s desirable for you.
Once you go down this road, any test dependencies you define, either on external libraries such as Scalacheck, Discipline or (naturally) Americium will be pulled into your published artifacts, unless you want to get into provided scopes in your SBT definition.
If all your tests are just a couple of lines of Python-style executable examples, then so be it, but be aware of the pitfalls.
Testing is a very complex and opinionated topic.
During my 10 years of experience, I have seen so many arguments about the “superior” methodology style… that at this point it is no longer funny.
Yes, this style can be very useful for certain situations, but it would be worse for others.
So, while I understand why you like it, and why you would like to use it in Scala. Just, please, remember that it is just your preference, not an objective truth.
Having said that, I do think it would be good to have at least one tool that supported this style. Both, because I do agree it has its use, and also just to allow folks who like it to be happy; which in turn could increase adoption.
However, the major problem with that style is that it requires either deep language or tooling support, compared to other testing styles.
As @sageserpent-open has pointed out, for this style to work, the test code and test dependencies must be able to remove themselves when publishing the real code. For libraries, if that is not possible, then the tool would be a big NO-NO, but even for applications, while not as critical, increasing your public API with unwanted functions and bundling unused dependencies can be problematic on multiple ways, security being a very tricky one.
I think your best bet, if you really wanted to do this, would probably be to do source preprocessing. You can hook into the mill sources task to create new sources on the fly and point at them instead of your original sources.
If you create something that is super-extra-easy to parse out, but otherwise can be read by some other testing framework, then you can have your regular compile task edit out your tests, but your test task compile them in and depend on that instead of your regular compilation.
You might need more boilerplate than simply an annotation in order to get the tests running easily. Although it’s possible to do some reasonably sophisticated inspection of semi-compiled code (e.g. with scalafix), having dumb string markers is sometimes a lot easier if you’re trying to hack together something without too much effort. So rather than a simple annotation, you’d have something like
The edit could be as simple as removing the */ at the end of the test line and /* at the start of the end test line; then you have a simple ifdef-like way to omit the code when actually compiling and not testing. Then you could generate a stub to get JUnit to call that when running the test action, or possibly just literally use JUnit annotations in the source file.
I don’t recommend it, but I think it wouldn’t be that difficult. It’s a “takes a few hours to set up and validate” kind of thing, if you otherwise know the tooling you’re using.
FWIW, I find that there are both benefits and drawbacks to the Rust-style in-source tests; enough of each so that I can make peace with just doing it the other way when using the other language.
Yes, sorry I did not mean to say that it is the superior style, just that there are compelling arguments in favor of it, and perhaps some compelling arguments against it, as testing is still an evolving practice in our industry and the jury is still out on which styles are generally better. I would like Scala to support it as one extra option for those who find it productive in their use cases.
Most JVM tools are directory based, so you probably won’t get any general purpose support for this test files layout in Scala land. But If you accept having the tests in files right next to the file under test, e.g. with a special extension or marker suffix (Foo.test.scala or FooTest.scala), you could separate these files very easily in Mill (via allSourceFiles), and just feed the right files to the relevant compiler.
I think the suffix way would be the right one. I would honestly hate a bit having everything in one file, it becomes quite unwieldy if you want to add a lot of tests. Though finding the right test is sometimes problematic, so we could just try to fix that with suffix.
This is already how Scala CLI does it, though it’s main purpose is smaller projects.
This does depend what other coding habits you are following in conjunction. I like to keep things very small and typically reserve each file for 1-class or a few functions at most, rarely exceeding 50-80 lines total, sometimes only a single function in a file. So there would still be space for tests at the bottom.
I acknowledge lots of programmers would hate this style also (lumpers vs splitters), but this style and in-source unit testing do go well together. Nice to have it as an option.