I would like to ask for some help automating a process which I don’t know how to do without the IntelliJ menu items. I would like to write a shell script (could also be a scala script but I think shell script is probably easier) will take some specified scala source files from several source locations, will then (1) “combine” the files, (2) run the code, and (3) examine the results.
(1) By combine, I’m not sure what I actually mean. One possibility is to create a tmp directory, and just copy the files there. Another possibility would be to concatenate the files into a single file but each file might lead with a package statement. Another possibility might be to create some sort of ?sbt? script which will simply reference the specific files from their given location. My intuition is that copying the relevant files into a single tmp directory and instructing the scala compiler to ONLY use files in that directory (plus the standard library) is the easiest and least error prone.
(2) By run the code I mean make sure the code compiles, and run the designated test suite, which is defined in one of the specified source files.
(3) By examine the results I mean to figure out if the tests passed and perhaps distinguish this from a case of compilation failed.
I am perfectly willing to provide additional information to the shell script such as name of test suite class. It is not necessary that the code auto-magically figure this out. And I’m perfectly willing to hand edit the script if the file names, locations, or class names change.
Why do I need this? I have given my students several template files of scala code with ??? in certain locations, and I’ve provided them with test suites. I also have written a solution and verified that the solution passes my internal test cases. foreach template for each student, I’d like to substitutive that student’s completed template for my solution, and then run the tests, thus determining whether the student did his homework. In each case the student template uses import to include some code provided in the lecture. For all the test suites and lecture code I want to be sure that MY version of the reference code is used. I’m happy to write a different unix script for each of the different templates, but I want the same script to then run for each student.
I’d be grateful if someone could help me develop this script.
I would put the templates into a Git repository that your students can clone. Students then hand in their work either by making a pull request or by handing in a diff. By checking out a student’s branch or applying their diff, you can run the tests.
Thanks for the suggestion, but the problem of distributing the templates is solved already. We are using a moodle (centralized at our engineering school). The problem is how to test whether each homework assignment submission of each students actually compiles and passes the tests and the students claim.
I have about 15 students and there will be some 20 homework assignments. So I could of course run them by hand by clicking on the little “Run Tests” Icon 20*15=300 times, but I’d prefer to do that from a script. The challenge is to write the script, and assure, by construction, that the only file that’s different per student is his single file submission.
BTW, we did consider using a git (gitlab) based solution. But I didn’t know enough about git and gitlab to know whether it was possible to distribute the files to students, and allow them to submit their work via push or pull request, but not allow other students to see anybody else’s submission.
OK, so I think I came up with a solution. I’ll explain it here for posterity, hoping that my experience might help someone in the future. Note that this is a UNIX-only solution.
I completely welcome any feedback for potential loopholes and how I can fix them. For example, I have done nothing to detect malicious code provided by students.
I wrote a shell script, grade.csh which takes command line arguments indicating
Each such invocation specifies
a) the student identifier
b) the name of the assignment
c) the Scala class name of the test suite
d) the number of tests to expect to pass
e) a path to the student-template.scala, the name of the file the student has modified and submitted.
f) a path to the file containing the test suite class definition
g) path names to zero or more files containing predefined code promised to the student to be available from the lecture
the script creates a directory in /tmp whose path is a function of the assignment name, and student id.
copies the mentioned .scala files into that directory respecting the src/{main,test}/scala directory structure
creates build.sbt with the following content, assignment name interpolated at $name
executes shell command: sbt compile "testOnly $suitename" |& sed -e 's/e[[][0-9]*m//g' > sbt.out
which substitutes the class name of the test suite for $suitename. The sed invocation removes the colorization which sbt tries to create, making it easier to parse the results.
the following grep command determines if the number of tests pass as expected, meaning the student’s code passes the required tests.
For each assignment I have written a separate shell script which invokes grade.csh with the information specific to that particular assignment.
SUMMARY: The key insights are:
0) redundantly install scala and sbt using brew as IntelliJ has its own version which the public can’t (easily) use.
copy the code to sanitary temporary directory
invoke sbt with compile and testOnly command line switches
parse output textually for number of tests passing
P.S., I can happily make the shell script available to anyone who is interested. But it is a bit of a hack for my personal use. The specific details will of course be useless to anyone else.