Americium - Test cases galore! Automatic case shrinkage! Bring your own test style. For Scala and Java

sageserpent-open · August 11, 2021, 7:50am

Introducing Americium: https://github.com/sageserpent-open/americium…

Offers automatic shrinkage to a minimal or nearly-minimal test case. Yes, invariants are preserved on test case data.
Shrinks efficiently.
Offers direct reproduction of a failing, minimised test case.
Covers finite combinations of atomic cases without duplication when building composite cases.
Gets out of the way of testing style - doesn’t care about whether the test are pure functional or imperative, doesn’t offer a DSL or try to structure your test suite.
Supports Scala and Java as first class citizens.
Supports covariance of test case generation in Scala, so cases for a subclass can be substituted for cases for a supertrait/superclass.
Supports covariance of test case generation in Java, so cases for a subclass can be substituted for cases for a superinterface/superclass.
Allows automatic derivation of test case generation for sum/product types (aka case class hierarchies) in the spirit of Scalacheck Shapeless.

It’s in early stages - one look at the introductory documentation will confirm that, but it’s reached the point where I’m going to attempt to use it seriously on my own work.

If others would like to give it a whirl, I’d love to hear their feedback.

The latest artifact is mentioned here: https://mvnrepository.com/artifact/com.sageserpent/americium_2.13/0.1.14-trials-rc

which reminds me to add a nice version widget to the GitHub front page!

If you’re a Scalacheck user, then where you would start with a Gen.const, Gen.oneOf, Gen.frequency, you cutover to:

a) Getting a TrialsApi via Trials.api on the Trials companion object: val api = Trials.api …

b) … then you hit that instance for your Trials … these are analogous to the Gen monad instances, so
api.only("One string and that's it").
api.choose(1, 3, -4),
api.alternateWithWeights(1 -> oneSortOfTrials, 5 -> aMorePopularSortOfTrials) …

c) … then you apply a limit on how many cases you want and hit your own favourite test lambda with it:
myGrandTrialsInstance.withLimit(500).supplyTo{testCase => ........... }

What you put in that lambda is up to you - the point is that it either executes successfully, or throws an exception.

You can see a simple example of usage at the top here:

https://github.com/sageserpent-open/americium/blob/master/src/test/scala/com/sageserpent/americium/Examples.scala

There are some Java smoke tests here too: https://github.com/sageserpent-open/americium/blob/master/src/test/scala/com/sageserpent/americium/java/TrialsApiTests.java

Hope you have fun with it!

sageserpent-open · August 12, 2021, 2:34pm

I forgot to mention - there are practically infinite streamed cases too, these are analogues to the Arbitrary.arbitrary* from Scalacheck:

api.integers // Arbitrary.arbInt
api.strings // Arbitrary.arbString
api.instants // There's one out there somewhere, or you rolled your own by mapping....

api.stream(myTransformThatTakesALongAndYieldsSomeCase) // For when you want to write one from scratch and control the definition of shrinking for your type.

sageserpent-open · September 6, 2021, 8:19am

A brief final update for now:

I have added some more documentation, I hope this makes it clearer what this is about and how to use it.
I did some more dog-food testing of the Java API and have streamlined it a little so that it is more on par with the Scala API.
Improved the test case shrinkage after making the unit tests for this more aggressive.

The latest version is here: https://mvnrepository.com/artifact/com.sageserpent/americium_2.13/0.1.17

May your failing test cases be minimal, and quick to reproduce…

sageserpent-open · January 4, 2022, 6:14pm

… and another update, this time to version 1.2.0:

I’ve been doing yet more dog-food testing of both the Java and Scala APIs, found a couple more bugs and deficiencies, written some new tests and fixed the problems. The APIs for both Scala and Java have been worked on some more, and these days I’m using it actively, so there is at least one satisfied customer out there.

Shrinkage has been improved, here’s the output of one of the tests to feast your eyes on - btw, I have no idea if those ideograms have any meaning in combination, so apologies in advance if there is any unintended undesirable language in there, Google translate didn’t reveal anything untoward:

Provoking case for 'Has more than one text item whose converted values sum to more than 7.': Vector(13012, 37024)
Provoking case for 'Has either four or five characters.': Vector(㿜, ঞ, 풍, 䮋)
Provoking case for 'Has either four or five characters - variation.': Vector(⑕, 鄷, 灍, )
Provoking case for 'Has either four or five characters - variation with shrinkable character range.': Vector(q, q, q, q)
Provoking case for 'Has either four or five characters - this used to be a pathologically slow example.': Vector(䠘, ҿ, 쁂, 텡)
Provoking case for 'Has more than one item and sums to more than 7.': Vector(0.0, 7.907022966713223)
Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than one item, no zeroes and sums to more than 7.': Vector(4, 4)
Provoking case for 'Has more than one item, at least one zero and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than one item, no zeroes and sums to more than 7.': Vector(3, 5)
Provoking case for 'Has more than one item, at least one zero and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than 7 items.': Vector(0, 0, 0, 0, 0, 0, 0, 0)
Provoking case for 'Has more than one item, at least one non-zero and sums to a multiple of 7.': Vector(0, 7)
Provoking case for 'Has more than one item, no zeroes and sums to a multiple of 7.': Vector(1, -1)
Provoking case for 'Has more than one item, at least one zero and sums to a multiple of 7.': Vector(0, 0)
Provoking case for 'Has more than one item and sums to a positive multiple of 7.': Vector(0, 7)
Provoking case for 'Has more than one item, no zeroes and sums to a positive multiple of 7.': Vector(3, 4)
Provoking case for 'Has more than one item, at least one non-zero and sums to a positive multiple of 7.': Vector(0, 7)
Provoking case for 'Flattened binary tree with more than one leaf that sums to a multiple of 19 greater than 19.': Vector(0, 38)
Provoking case for 'Flattened binary tree with more than one leaf and no zeroes that sums to a multiple of 19 greater than 19.': Vector(-1, 39)
Provoking case for 'Flattened binary tree with more than one leaf and at least one zero that sums to a multiple of 19 greater than 19.': Vector(38, 0)
Provoking case for 'Has more than five items, at least one non-zero and sums to a multiple of 7.': Vector(1, -1, 0, 0, 0, 0)
Provoking case for 'Has more than five items and sums to a multiple of 7 less than -7.': Vector(0, 0, -14, 0, 0, 0)
Provoking case for 'Has more than two items and is not sorted in ascending order.': Vector(1, 0, 0)
Provoking case for 'Has more than two items and no duplicates.': Vector(1, -1, 0)
Provoking case for 'Flattened binary tree with more than two leaves whose odd-indexed leaves contain zeroes and even-indexed leaves contain non-zero values that sum to a multiple of 31 greater than 31.': Vector(46, 0, 16)
Provoking case for 'List with more than two elements whose odd-indexed elements contain zeroes and even-indexed elements contain non-zero values that sum to a multiple of 31 greater than 31.': Vector(22, 0, 40)

I shall take a break for now - so what we have is purely for Scala 2.13 (and to some extent Scala 3 via cross compatibility, although the Magnolia automatic derivation support may not work in Scala 3). Having said that, if anyone fancies taking a crack at cross-version publishing, I’d be receptive to a pull-request / mentoring - I did make some efforts in this direction for 2.12 and 3, but it didn’t look trivial.

For those like me who moonlight in Java too, the Scala language version isn’t so important, so it’s pretty much fully cooked for that crowd. One possibility is to publish a shaded JAR, which would make its adoption by Scala 3 users a little easier. Again, I’m open to pull requests / an issue being raised if there is interest.

Enjoy!

sageserpent-open · January 31, 2022, 10:18am

Update to version 1.1.2…

I revisited the Scala 3 port and sorted it out, so now there is a cross Scala version build, and thus artifacts in Maven for Scala 2.13 and Scala 3.

Enjoy!

sageserpent-open · August 3, 2022, 6:48am

Release 1.8.1 is out. Since 1.2.0, the main changes have been:

Bug fixes
Improvements to the quality of test case shrinkage
Integration with JUnit5 (not of so much interest if you are 100% Scala focussed, by if you work in Java too…)
More convenient overloads for supplying scalar values within bounds, and with custom shrinkage target values
More control over how many test cases are supplied - can now use a timed strategy, a Scalacheck-style discard ratio strategy or roll your own completely. Strategies can vary depending on whether the test is running in ‘discovery mode’ to find a first failure, or is shrinking an existing failing case.

https://github.com/sageserpent-open/americium/releases

Americium was kindly accepted into Johannes Link’s shrinking challenge repository, so you can see it slugging it out with other parameterised testing frameworks!

https://github.com/jlink/shrinking-challenge

Enjoy!

sageserpent-open · February 8, 2023, 6:52pm

Following on … I’m working on Release 1.10.0 which I hope to leave as a definitive, stable version while I do other things. While preparation for that goes on, I would be interested in whether anyone who looked at the README: https://github.com/sageserpent-open/americium found it clear as to what’s on offer by Americium and how it can be used.

I have no idea as to the uptake of this library other than in some commercial contexts where I have interested colleagues enough in it (and property based testing in general) for it to be adopted. There are very few issues raised on GitHub from other people, so either everyone thinks it’s fabulous or there is not much adoption.

If you did look at it a while back (or even from reading this update) and found it all a bit cryptic, do let me know where you had problems with it, it would be good to to address any pain points with the documentation.

I’m also curious as to whether folks thought “Meh, I use Scalacheck etc already, not interested” as that gives me some idea as to whether the shrinkage offering is really something others are concerned about in the context of property-based testing.

Thanks in advance!