Can autograder be cheated?

  by   Cécilia Pradic






Departments Computer Science
DescriptionA number of modules in the CS department use csautograder.swansea.ac.uk that allows to automate part of the grading of programming homework. This system is a deployment of Autograder.io (https://eecs-autograder.github.io/autograder.io/) which allows lecturers to set up homework in a bespoke way. The lecturers typically write the homework and some code for testing the coursework submission themselves, and then the system run the code provided by the lecturer with an access to an individual student submission in a sandboxed environment (also provided by the lecturer, although those can be relatively standard). While the sandboxing and the framework make it safe to assume that malicious submissions would be unlikely to harm the overall system, whether the tests and the sandbox environment are 1) correct and 2) secure from being manipulated is left to the lecturers. Having 1) and 2) compromised could mean that the automated grading does not award correct marks. The goal of this project would be to investigate what vulnerabilities could exist in those individual setups that would allow to engineer submissions that get full automated marks without actually solving the coursework questions, and propose some countermeasures less time-consuming than investigating the submission history of a particular user. I would be happy to provide some sample grading setups that I wrote in the past to be put to test, and for an attack on those be documented.
PreparationProbably things related to cybersecurity would be helpful - knowing that injection attacks exist sure is helpful since you would be designing some attacks. I do not have experience personally, but am familiar enough with the tests I do write to help point out some things to try. It is possible you might need/want to deploy a local instance of autograder on your computer, so doing that in advance might be nice.
Project Categories January Cohort, Security
Project Keywords Cybersecurity, Testing and Verification, Web Applications


Level of Studies

Level 6 (Undergraduate Year 3) yes
Level 7 (Masters) yes
Level 8 (PhD) yes