Americium - Test cases galore! Automatic case shrinkage! Bring your own test style. For Scala and Java

Introducing Americium: https://github.com/sageserpent-open/americium

  1. Offers automatic shrinkage to a minimal or nearly-minimal test case. Yes, invariants are preserved on test case data.
  2. Shrinks efficiently.
  3. Offers direct reproduction of a failing, minimised test case.
  4. Covers finite combinations of atomic cases without duplication when building composite cases.
  5. Gets out of the way of testing style - doesn’t care about whether the test are pure functional or imperative, doesn’t offer a DSL or try to structure your test suite.
  6. Supports Scala and Java as first class citizens.
  7. Supports covariance of test case generation in Scala, so cases for a subclass can be substituted for cases for a supertrait/superclass.
  8. Supports covariance of test case generation in Java, so cases for a subclass can be substituted for cases for a superinterface/superclass.
  9. Allows automatic derivation of test case generation for sum/product types (aka case class hierarchies) in the spirit of Scalacheck Shapeless.

It’s in early stages - one look at the introductory documentation will confirm that, but it’s reached the point where I’m going to attempt to use it seriously on my own work.

If others would like to give it a whirl, I’d love to hear their feedback.

The latest artifact is mentioned here: https://mvnrepository.com/artifact/com.sageserpent/americium_2.13/0.1.14-trials-rc

  • which reminds me to add a nice version widget to the GitHub front page!

If you’re a Scalacheck user, then where you would start with a Gen.const, Gen.oneOf, Gen.frequency, you cutover to:

a) Getting a TrialsApi via Trials.api on the Trials companion object: val api = Trials.api

b) … then you hit that instance for your Trials … these are analogous to the Gen monad instances, so
api.only("One string and that's it").
api.choose(1, 3, -4),
api.alternateWithWeights(1 -> oneSortOfTrials, 5 -> aMorePopularSortOfTrials)

c) … then you apply a limit on how many cases you want and hit your own favourite test lambda with it:
myGrandTrialsInstance.withLimit(500).supplyTo{testCase => ........... }

What you put in that lambda is up to you - the point is that it either executes successfully, or throws an exception.

You can see a simple example of usage at the top here:

https://github.com/sageserpent-open/americium/blob/master/src/test/scala/com/sageserpent/americium/Examples.scala

There are some Java smoke tests here too: https://github.com/sageserpent-open/americium/blob/master/src/test/scala/com/sageserpent/americium/java/TrialsApiTests.java

Hope you have fun with it!

3 Likes

I forgot to mention - there are practically infinite streamed cases too, these are analogues to the Arbitrary.arbitrary* from Scalacheck:

api.integers // Arbitrary.arbInt
api.strings // Arbitrary.arbString
api.instants // There's one out there somewhere, or you rolled your own by mapping....

api.stream(myTransformThatTakesALongAndYieldsSomeCase) // For when you want to write one from scratch and control the definition of shrinking for your type.

A brief final update for now:

  1. I have added some more documentation, I hope this makes it clearer what this is about and how to use it.
  2. I did some more dog-food testing of the Java API and have streamlined it a little so that it is more on par with the Scala API.
  3. Improved the test case shrinkage after making the unit tests for this more aggressive.

The latest version is here: https://mvnrepository.com/artifact/com.sageserpent/americium_2.13/0.1.17

May your failing test cases be minimal, and quick to reproduce…

1 Like

… and another update, this time to version 1.2.0:

I’ve been doing yet more dog-food testing of both the Java and Scala APIs, found a couple more bugs and deficiencies, written some new tests and fixed the problems. The APIs for both Scala and Java have been worked on some more, and these days I’m using it actively, so there is at least one satisfied customer out there.

Shrinkage has been improved, here’s the output of one of the tests to feast your eyes on - btw, I have no idea if those ideograms have any meaning in combination, so apologies in advance if there is any unintended undesirable language in there, Google translate didn’t reveal anything untoward:

Provoking case for 'Has more than one text item whose converted values sum to more than 7.': Vector(13012, 37024)
Provoking case for 'Has either four or five characters.': Vector(㿜, ঞ, 풍, 䮋)
Provoking case for 'Has either four or five characters - variation.': Vector(⑕, 鄷, 灍, )
Provoking case for 'Has either four or five characters - variation with shrinkable character range.': Vector(q, q, q, q)
Provoking case for 'Has either four or five characters - this used to be a pathologically slow example.': Vector(䠘, ҿ, 쁂, 텡)
Provoking case for 'Has more than one item and sums to more than 7.': Vector(0.0, 7.907022966713223)
Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than one item, no zeroes and sums to more than 7.': Vector(4, 4)
Provoking case for 'Has more than one item, at least one zero and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than one item and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than one item, no zeroes and sums to more than 7.': Vector(3, 5)
Provoking case for 'Has more than one item, at least one zero and sums to more than 7.': Vector(8, 0)
Provoking case for 'Has more than 7 items.': Vector(0, 0, 0, 0, 0, 0, 0, 0)
Provoking case for 'Has more than one item, at least one non-zero and sums to a multiple of 7.': Vector(0, 7)
Provoking case for 'Has more than one item, no zeroes and sums to a multiple of 7.': Vector(1, -1)
Provoking case for 'Has more than one item, at least one zero and sums to a multiple of 7.': Vector(0, 0)
Provoking case for 'Has more than one item and sums to a positive multiple of 7.': Vector(0, 7)
Provoking case for 'Has more than one item, no zeroes and sums to a positive multiple of 7.': Vector(3, 4)
Provoking case for 'Has more than one item, at least one non-zero and sums to a positive multiple of 7.': Vector(0, 7)
Provoking case for 'Flattened binary tree with more than one leaf that sums to a multiple of 19 greater than 19.': Vector(0, 38)
Provoking case for 'Flattened binary tree with more than one leaf and no zeroes that sums to a multiple of 19 greater than 19.': Vector(-1, 39)
Provoking case for 'Flattened binary tree with more than one leaf and at least one zero that sums to a multiple of 19 greater than 19.': Vector(38, 0)
Provoking case for 'Has more than five items, at least one non-zero and sums to a multiple of 7.': Vector(1, -1, 0, 0, 0, 0)
Provoking case for 'Has more than five items and sums to a multiple of 7 less than -7.': Vector(0, 0, -14, 0, 0, 0)
Provoking case for 'Has more than two items and is not sorted in ascending order.': Vector(1, 0, 0)
Provoking case for 'Has more than two items and no duplicates.': Vector(1, -1, 0)
Provoking case for 'Flattened binary tree with more than two leaves whose odd-indexed leaves contain zeroes and even-indexed leaves contain non-zero values that sum to a multiple of 31 greater than 31.': Vector(46, 0, 16)
Provoking case for 'List with more than two elements whose odd-indexed elements contain zeroes and even-indexed elements contain non-zero values that sum to a multiple of 31 greater than 31.': Vector(22, 0, 40)

I shall take a break for now - so what we have is purely for Scala 2.13 (and to some extent Scala 3 via cross compatibility, although the Magnolia automatic derivation support may not work in Scala 3). Having said that, if anyone fancies taking a crack at cross-version publishing, I’d be receptive to a pull-request / mentoring - I did make some efforts in this direction for 2.12 and 3, but it didn’t look trivial.

For those like me who moonlight in Java too, the Scala language version isn’t so important, so it’s pretty much fully cooked for that crowd. One possibility is to publish a shaded JAR, which would make its adoption by Scala 3 users a little easier. Again, I’m open to pull requests / an issue being raised if there is interest.

Enjoy!