I guess you mean by that, just a sub-shell controlled from the shell script?
In the mean time, I edited the student’s code and fixed the infinite loop. Turns out several students made the same mistake, so it’s good that I understand the error so I can explain the gocha during the next lecture.
I mean, you may do this to at least reduce the amount of not test time.
> sbt test:compile // Download dependencies, compile the source code, and compile the tests code.
> timeout 60 sbt compile "testOnly SuiteClassName" >& sbt.out
You may even use bloop instead of sbt which will reduce the start footprint.
But, I guess the best would be to write your own wrapper over Scalatest to ensure the correct timeout, I believe that was what the Coursera course did.
I would also like to run this in a container, docker for example, because at the moment I’m running student code on my local laptop, which is a security risk. none of my students are malicious, but a real solution would be much more secure than my home-grown one.
TimeLimits and TimeLimitedTests from ScalaTest are easy to use. They can also interrupt a thread if a test takes too long. I use a setup where I assign the same time limit to all “short” tests and to all “long” tests (using tag objects Slow and Fast).
The real difficulty comes from threads that refuse to terminate, as in students’ infinite loops. I’ve set up my framework to run tests in separate, “stoppable” threads (using the Java stop method despite its deprecation). It’s a little tricky, but it’s been running reliable for several semesters now.
I have a few other goodies (to calculate a weighted grade from passed/failed tests, to catch stack overflows, to test multi-threaded programs, …), nothing big. I use the framework in two classes I teach (one is in Scala; the other is in Java, but I write my grading tests in Scala). I should cleanup the code and put it on GitHub.
From the documentation, it seems I need to modify every test suite to also inherit from TimeLimitedTests. I was hoping for a command line based solution. For the moment, I’ll stick with the timeout solution I’ve outlined above.