socrates is a Python program that was written to make the grading process more efficient by automating the repetitive tasks ordinarily done by human graders. To this end, the program provides tools to run pre-programmed tests defined by an instructor or the teaching staff in a criteria file. The criteria file is a plain text file that describes the required parts of a submission, and any number of tests designed by the instructor or teaching staff. Reading this file, socrates will determine which parts of a submission are present, and run the tests for the parts of a submission that are present. Missing parts of a submission or failed tests constitute the deductions for a particular submission. When socrates has run all the tests it can, it writes a grade file to the student's submission directory.

Obviously, there are some cases in which a test for socrates cannot be designed. For example, examining a solution for proper code style or appropriate documentation must be done by a human grader. For this task, the review test was created. When socrates finds a review test, it pauses the grading to allow the human grader to review a part of the submission and take deductions, if necessary.

The ultimate goal is to automate as many tests as possible. However, socrates is a relatively new tool, and currently only has support for a few kinds of assignments.

For much more information, see our wiki on GitHub.