Algorithmic Reasoning Tasks: A Novel Approach to Assess and Predict Programming Abilities of Novice Learners

Core Concepts

Algorithmic Reasoning Tasks (ARTs) can effectively assess the reasoning skills required for code writing and be used to predict student performance in introductory programming courses.

Abstract

The study presents a novel approach to assess and predict the programming abilities of novice learners through Algorithmic Reasoning Tasks (ARTs). The key highlights are: The ART framework includes three types of questions - Detection, Comparison, and Analysis - that require relational-level reasoning beyond just code tracing. The ART instruments can be automatically assessed, unlike manual "explain in plain English" tasks used in prior work. The study used machine learning models, particularly Random Forest, to predict student performance on code writing tasks based on their performance on ART questions. The Random Forest model achieved an accuracy of 85.45% in predicting student success in code writing, outperforming Logistic Regression. The ART Comparison questions were found to have the highest feature importance in the Random Forest model, indicating their strong correlation with code writing abilities. Compared to prior approaches like Activity Diagrams and Parson's Puzzles, the ART instruments showed higher Pearson correlation with code writing performance. The study suggests that ART instruments can form a learning trajectory to gradually develop the reasoning skills needed for effective code writing, helping address the high failure rates in introductory programming courses.

Stats

The ART Detection question required students to extract the overall purpose of the algorithm and apply it to different input arrays. The ART Comparison question required students to identify algorithms with similar behavior. The ART Analysis question required students to reason about the algorithm's performance characteristics.

Quotes

"Many students in introductory programming courses fare poorly in the code writing tasks of the final summative assessment." "To extend this work to larger groups, we have devised several question types with varying cognitive demands collectively called Algorithmic Reasoning Tasks (ARTs), which do not require manual marking." "Our preliminary research suggests ART type instruments can be combined with specific machine learning models to act as an effective learning trajectory and early prediction of code-writing skills."

Key Insights Distilled From

Creating a Trajectory for Code Writing

by Shruthi Ravi... at arxiv.org 04-04-2024

https://arxiv.org/pdf/2404.02464.pdf

Deeper Inquiries

How can the ART framework be further extended to develop a comprehensive learning trajectory for programming skills, beyond just code writing?

The ART framework can be extended by incorporating a wider range of tasks that target different aspects of programming skills. For example, tasks focusing on problem analysis, solution planning, and coding can be included to cover the entire process of solving programming problems. Additionally, tasks that require debugging, optimization, and refactoring can be integrated to enhance students' skills in these areas. By diversifying the types of tasks within the ART framework, a more comprehensive learning trajectory can be developed to address various facets of programming skills development.

How can the ART instruments be integrated into the curriculum to provide personalized feedback and support for struggling novice programmers?

To integrate ART instruments into the curriculum effectively, instructors can incorporate them into regular assessments throughout the course. By including ART tasks in formative assessments, instructors can track students' progress and provide timely feedback on their relational thinking and problem-solving skills. Additionally, utilizing automated grading systems for ART tasks can streamline the feedback process and allow for personalized insights into each student's strengths and areas for improvement. Instructors can use the data from ART assessments to identify struggling novice programmers and offer targeted support, such as additional resources, one-on-one sessions, or remedial assignments tailored to address specific skill gaps.

What other machine learning techniques could be explored to enhance the predictive power of the ART-based approach?

In addition to Random Forest and Logistic Regression, other machine learning techniques could be explored to enhance the predictive power of the ART-based approach. Support Vector Machines (SVM) and Neural Networks are two powerful algorithms that could be applied to analyze the relationship between students' performance on ART tasks and their code writing abilities. SVM can handle complex relationships in the data, while Neural Networks can capture nonlinear patterns that may exist in the dataset. Ensemble methods like Gradient Boosting and AdaBoost could also be considered to combine multiple models and improve prediction accuracy. By experimenting with a variety of machine learning techniques, the ART-based approach can be further refined to provide more accurate predictions and insights into students' programming skills.