Does anyone have experience with auto-grading scala code on a “moodle”.
This is an open source learning platform, what is vast of the things it can do. Our school uses one and it is possible for teachers to set up programming assignments to students, who submit their solutions to have them auto graded. Ours is set up for Python and several other languages, but not for Scala. The administrators of our site are not Scala experts but are happy to set it up for Scala. I don’t really know what such a setup even ought to look like. What all must be installed behind the scenes? How do I (as instructor) need to set up the assignments so they can be auto graded.
Currently I’m planning (for each assignment) to give the students a template scala file with ??? in various places which the student has to complete. I’m also going to give the students a basic test suite file, which uses import homework._ to import their code. I’d like the auto-grader to load their code (the file the student submitted) and compile it with the test suite which I provide plus other files as necessary which I have to specify.
If anyone can give me insight as to how this is supposed to work, I can explain the use model to our system administrators.
For example, do I need to write a UNIX shell script for each assignment which calls the scala compiler from the unix command line specifying the full paths of all the files, and then runs the program. I don’t know how to run the test suites from the unix command line and check for success or failiure… From IntelliJ I just select the name of the class and select “Run Asisgnment1TestSuite” for example.
I know nothing about Moodle (aside from the fact that several of my friends who do instructional design professionally use it a lot). But from the sound of things, I’d strongly recommend getting more comfortable working in sbt instead of IntelliJ.
For our own courses, we set up a small sbt project for each module, typically with all of the code in the “test” side of the tree. As you say, stuff is often stubbed out with ???; as I’ve mentioned before, we also sometimes make use of should compile and shouldNot compile. We do require students to install sbt before starting the course.
With that setup, all you need from the command line is to say sbt test, and it’ll either succeed or fail. We don’t do automated checking, so I’m not certain whether the resulting success/failure propagates out as an error code, but I think it does.
That involves a little boilerplate per exercise set, for the sbt project. But most of that is copy-paste (or say “sbt new” to generate the skeleton), and this approach works just like ordinary Scala programs, with no mucking about with custom scripts and such. So I recommend it if it’s possible.
The only downside is that you can’t really think of this just as a single isolated file; each exercise set really is a directory structure. That’s realistic: I find that some of my more interesting lessons really want the student to be working in anywhere up to half a dozen files, mimicking what the real Scala code would look like. But I don’t know if it works for your automated setup…
So what is the concept behind sbt based auto-grading? Does sbt run by the student need to connect to a server (controlled by the instructor) and send some files, which the server “runs” records a grade? In what sense is this capability built in to sbt, and how much would need be be implemented for the particular use case?
I wrote my own little server for doing auto-grading of student Scala code. I run their code in a JVM with some restrictions. It means I can’t test certain things, but it works for my purposes. We also use Moodle, but trying to get into that would mean working with other parts of the University that i think would make my life a lot harder. I just host my stuff on one of the CS machines.
I actually wouldn’t fear it that much. It’s a lot like Scala itself, in a sense: folks often try to learn all of it and run screaming. But learning enough for day-to-day use isn’t actually hard, especially now that sbt new is a thing – you generally only need a small fraction of its power for smaller projects.
That said,
We don’t actually have auto-grading, since we don’t have grading. (We do corporate training, not academic, so the needs are very different.) So this was just a suggestion of how you might make command line invocation work, not a fully-baked solution…
I didn’t know that Moodle was offering that kind of plugin – we also set up our own grading system from the ground up, using a dedicated submission web interface, a dedicated maven script to compile student projects (exported from ScalaIDE), and a home-made script to extract results of unit tests and scalastyle messages.
I looked up the plugin (I think) you mention, and it seems to be documented here https://coderunner.org.nz/
@jimka My recommendation is to use bloop (which has btw a CLI) and allows to compile, test and run Scala programs. You can write your grading architecture on top of it, be it with writing CLI commands or if you do it programmatically in Scala/Java via BSP. You can read more about it in our integration guide Integrate with the Build Server · Bloop
My setup is to have an sbt file for each assignment/project (usually the same) and to write tests using Scalatest + a few extensions of my own (e.g., to get a grade based on which tests passed/failed and to deal with threads since I tend to have many assignments that involve concurrency). sbt handles dependencies (to Scalatest and my own extensions) and VM configuration (e.g., extra memory). Students push code to a Git server. My scripts pull from there and run on a dedicated computer.