I’m looking for suggestions of how to manage a group of about 50 students who need to turn in assignments which are scala programs which I need to grade. I have not 100% determined the grading criteria, but I assume there will be some account for (1) whether the program compiles (2) whether it runs on a set of secret test cases, and (3) perhaps in some cases performance calculation based on a range of different input sizes.
My current idea is that each student submit a single file named according to his student-id-number and assignment number. The file is required to implement a certain interface, but not the main program. I’ll have a wrapper program which will either be an extra file which will import their class, or I’ll have a script which simply wraps their file with a prolog and epilog.
I’d be happy to hear suggestions from people. Is there already a framework I can use for this which is not protected by paywall or copyright (the engineering school is for-profit, so in a sense I’d be using any such existing software for-profit).
Questions I don’t know how to address at this point is,
Is it possible to run students code in my environment, this seems very insecure, what if the students code accidentally tries to remove my home directory?
Are there library version issues I need to understand? I.e., do I have to care about which version of scala and various libraries I’m using or which the students are using?
Is a single file student-submission sufficient? Or do I need to accommodate multiple submissions, and thus have to support some sort of build-script nightmare?
Is there an infrastructure where each student can submit a single file thousands of times before a deadline, no student and intentionally or accidentally see or modify another student’s submission, and thereafter I can easily loop through all the files in batch? Students submitting attachments via email seems like the wrong solution.
Does Scala have naming conventions for files which prohibit the students from naming their file with their student id number?
What about file encoding? The students are mostly French. What if they use special french characters in the file or in their code?
For our course which uses Scala programs, we use a system provided by our school, which lets students submit solutions via SVN (each student gets his own repository). The system then optionally runs unit tests on the submissions and shows the result to the student. Resubmission is possible until a deadline. Sadly the code for it doesn’t seem to be available publicly. But from my experiences with it, I think I can still answer some of your questions.
You can migitate that danger by sandboxing the student program. There are different tools for different operating systems to do that, I’ve been using firejail (Linux) for when I had to run student code on my own machine (the submission system also uses some kind of sandboxing to run the tests).
Usually, you should have no problem if the minor version matches, so if your students all use 2.12.x,
the x should not matter (as long as you use the most current version yourself). As we had some additional requirements (FP course, using kind-projector compiler plugin), we provided an sbt file to all students. Their code was required to compile with that template. When testing the code, we used this template file everytime, ignoring any buildfiles the students may have submitted. This made submissions with several files pretty straigtforward.
Before the current version of our grading system, which uses a repository for submission anyway, we required the submission to be an archive with the correct directory stucture, this should not be too dificult for students.
Before our current system, we used a file submission form in a Moodle instance provided by our school. I know the people developing the current system and will ask, if they plan to opensource it.
In Scala, there are no restrictions on the file name, a numeric name is fine.
I’m not sure if this problem exists with Scala, but with Java we often had trouble with the compiler choking on encoding (german umlauts in comments). I just ran a script, that tried to compile the code, and if it failed it retried with -encoding Cp1252 added (which was the only encoding besides UTF-8 that we received, because it seems to be default in Eclipse on German Windows).
I wrote my own code for doing this type of thing, but I have also used HackerRank school (https://www.hackerrank.com/products/school/). For security, I provide a policy file that the JVM uses to limit what students can do. This only works because I never have them read files, use the network, etc. in the problems that I’m giving them.
You will need multiple submissions per student and the better your error reporting is when thing don’t go well, the better. To prevent students from abusing my server and to get them to write their own test code, after a few years I started taking points off if they submit more than a certain number of times.
The system I wrote would force the students to use a particular name for the primary code that I’m going to run.
One of my colleagues has considered trying to implement a system that uses CI testing on student submissions to GitHub. That could also be a possibility.
Our MOOCs run on the Coursera and edX learning management systems (LMS), which handle all the user authentication part. When a student submits his work to the LMS, he uploads his bytecode, which is then added to the classpath of our graders so that we can run our test suite on their work.
Our infrastructure is currently close-source, mainly because we use a mono-repo that also contains the solution of the assignments. But we could consider open sourcing some of our generic components, if that’s useful to other organizations…
We run the student’s code in a virtual machine.
In our case, our grading infrastructure contains the tests (which are unknown to the students), which are then linked against the student’s submission. Our graders are strongly coupled with the binary API of the students code. This means that the students have to use the same Scala major version than our graders.
We used to have an infrastructure that used the sources of the students’ work (as opposed to using binaries). That model only forced source compatibility between graders and students’ work, but that flexibility came with a price: grading times were longer (we had to start sbt and compile the student’s submission).
Most of our assignments fit into a single file, but some of them use several files (like real Scala projects).
OpenEdX is an open-source learning management system that does that. It seems to be a complex beast, though. GitHub Classroom could be an alternative.
Yes, I think that could be useful. Probably not for me, as my class starts within a month. But if I teach the class again in upcoming semesters, your suggestion could be interesting indeed.
(which was the only encoding besides UTF-8 that we received, because it seems to be default in Eclipse on German Windows).