TimeLimits and TimeLimitedTests from ScalaTest are easy to use. They can also interrupt a thread if a test takes too long. I use a setup where I assign the same time limit to all “short” tests and to all “long” tests (using tag objects Slow and Fast).
The real difficulty comes from threads that refuse to terminate, as in students’ infinite loops. I’ve set up my framework to run tests in separate, “stoppable” threads (using the Java stop method despite its deprecation). It’s a little tricky, but it’s been running reliable for several semesters now.
I have a few other goodies (to calculate a weighted grade from passed/failed tests, to catch stack overflows, to test multi-threaded programs, …), nothing big. I use the framework in two classes I teach (one is in Scala; the other is in Java, but I write my grading tests in Scala). I should cleanup the code and put it on GitHub.