I’m grading students submissions by using sbt from within a shell script.
The command line for sbt is something like the following.
> sbt compile "testOnly SuiteClassName" >& sbt.out
Is there a way to tell sbt to timeout if the tests take longer that some given threshold? Or do I need to handle this from the shell by sending a kill signal after waiting some number of seconds?
You could run the student code on a Thread that you kill after a certain time.
ScalaTest has some timeout for tests, although I’ve never used it.
I guess you mean by that, just a sub-shell controlled from the shell script?
In the mean time, I edited the student’s code and fixed the infinite loop. Turns out several students made the same mistake, so it’s good that I understand the error so I can explain the gocha during the next lecture.
one option which seems not so bad is the “timeout” command which is available on the mac from brew install coreutils.
> timeout 60 sbt compile "testOnly SuiteClassName" >& sbt.out
which will kill sbt after 60 seconds, if it is still running.
The thing is, it will count the time of sbt starting, tests and code compiling and the Scalatest framework init time. Not just your tests.
@BalmungSan, You are right. And the first time I run it it takes a long time to download all the libraries.
It would be nice to have a better solution.
I mean, you may do this to at least reduce the amount of not test time.
> sbt test:compile // Download dependencies, compile the source code, and compile the tests code.
> timeout 60 sbt compile "testOnly SuiteClassName" >& sbt.out
You may even use bloop instead of sbt which will reduce the start footprint.
But, I guess the best would be to write your own wrapper over Scalatest to ensure the correct timeout, I believe that was what the Coursera course did.
No, I was assuming your student’s code runs on a Thread in the JVM.
That would require expertise which I do not have.
I would also like to run this in a container, docker for example, because at the moment I’m running student code on my local laptop, which is a security risk. none of my students are malicious, but a real solution would be much more secure than my home-grown one.
TimeLimits and TimeLimitedTests from ScalaTest are easy to use. They can also interrupt a thread if a test takes too long. I use a setup where I assign the same time limit to all “short” tests and to all “long” tests (using tag objects Slow and Fast).
The real difficulty comes from threads that refuse to terminate, as in students’ infinite loops. I’ve set up my framework to run tests in separate, “stoppable” threads (using the Java stop method despite its deprecation). It’s a little tricky, but it’s been running reliable for several semesters now.
I have a few other goodies (to calculate a weighted grade from passed/failed tests, to catch stack overflows, to test multi-threaded programs, …), nothing big. I use the framework in two classes I teach (one is in Scala; the other is in Java, but I write my grading tests in Scala). I should cleanup the code and put it on GitHub.
I’ve started to explore that, but I don’t have a working solution yet. I use a Docker image I built (Docker) that contains Java and sbt. I can compile and run student code in it. It’s the deployment setup I don’t have.
From the documentation, it seems I need to modify every test suite to also inherit from TimeLimitedTests. I was hoping for a command line based solution. For the moment, I’ll stick with the timeout solution I’ve outlined above.