Core Concepts
Developing tools to cluster and select diverse programming solutions from MOOC submissions to present to students, in order to improve their learning experience.
Abstract
The paper presents a novel problem of efficiently processing and presenting diverse programming solutions to students in Massive Open Online Courses (MOOCs). The authors developed two tools to address this problem:
Adaptation of the existing plagiarism detection tool JPlag:
JPlag uses a greedy string tiling algorithm to calculate the distance between student submissions and cluster them.
However, JPlag has limitations in handling short Python submissions, processing only 5.3% of the tasks in the provided dataset.
Development of a new tool called Rhubarb:
Rhubarb first standardizes the submissions by applying 12 code transformations to bring algorithmically similar solutions to a common form.
It then calculates the structure-aware edit distance between the standardized solutions using the GumTree tool and applies hierarchical agglomerative clustering.
Finally, Rhubarb selects one representative example from each of the largest clusters, considering the code quality using the Hyperstyle tool.
The authors compared the default platform approach, JPlag, and Rhubarb on a set of 59 tasks. Eight experts rated the selected solutions based on diversity, code quality, and usefulness. The default platform approach received an average score of 3.12 out of 5, JPlag - 3.77, and Rhubarb - 3.50.
Since JPlag could only fully process 5.3% of the tasks, the authors implemented a system that uses JPlag on the tasks it can handle and Rhubarb on the remaining 94.7%.
The key contributions of this work are:
Rhubarb, a clustering tool for creating diverse solutions in MOOCs, which handles all the studied platform's tasks.
A library of code transformations for Python that can be useful in other applications.
An evaluation of Rhubarb with eight experts on 59 real tasks from a large MOOC platform, showing that it outperforms the default platform approach.
Stats
JPlag could fully process only 46 out of 867 studied tasks (5.3%).
JPlag partially processed 434 more tasks (50.1%), skipping some solutions.
JPlag did not process 387 tasks at all (44.6%).
Rhubarb successfully processed 100% of the 867 tasks.
Quotes
"To solve this novel problem, we adapted the existing plagiarism detection tool JPlag to Python submissions on Hyperskill, a popular MOOC platform. However, due to the tool's inner algorithm, it fully processed only 46 out of 867 studied tasks."
"Rhubarb was able to handle all 867 tasks successfully."