Automated Assessment of College-Level Physics Lab Reports Using Analytic Rubrics

Core Concepts

An end-to-end neural architecture, VerAs, can effectively automate the assessment of college-level physics lab reports using analytic rubrics, outperforming multiple baselines.

Abstract

This paper presents VerAs, an end-to-end neural architecture for the automated assessment of college-level physics lab reports using analytic rubrics. The key challenges addressed are:

Analytic rubrics can have diverse scoring strategies across different dimensions, making it difficult to automate the assessment process.
Identifying the relevant sentences in a report for each rubric dimension is a challenging task, even for human raters.

To address these challenges, VerAs has two main modules:

The Verifier module determines whether a report contains any content relevant to a given rubric dimension. It uses a dual encoder to learn effective similarities between report sentences and rubric dimensions.
The Grader module assesses the relevant sentences selected by the Verifier, using another dual encoder to capture the global context of the report. It is trained with an ordinal log loss objective to better handle the ordinal nature of the scoring scale.

The authors evaluate VerAs on two sets of college-level physics lab reports, and show that it outperforms multiple baselines, including OpenQA systems and Automated Essay Scoring (AES) approaches. Ablation studies demonstrate the importance of both the Verifier and Grader modules.
Additionally, the authors test the generalizability of the Verifier module on a dataset of middle school physics essays, where it performs well compared to other models.
The key findings are:

VerAs outperforms baselines on total report score metrics like MSE, Krippendorff's alpha, and weighted accuracy.
VerAs also performs better than baselines on the average per-dimension Spearman correlation and agreement.
The Verifier module plays an important role, despite the lack of labeled data for the sentence selection task.
The ordinal log loss objective in the Grader module is beneficial for the scoring task.
The Verifier module generalizes well to the middle school physics essay dataset.

Stats

The dataset consists of two sets of college-level physics lab reports, one on the behavior of a pendulum and the other on Newton's Second Law. Each report has an analytic rubric with 7 and 8 dimensions, respectively, scored on a 6-point scale.
The middle school dataset consists of responses to two essay prompts on the physics of roller coasters, with 6 and 8 main ideas to be identified in the essays.

Quotes

"With an increasing focus in STEM education on critical thinking skills, science writing plays an ever more important role in curricula that stress inquiry skills."
"Manual assessment can be slow, and difficult to calibrate for consistency across all students in large classes."
"VerAs first verifies whether a report contains any content relevant to a given rubric dimension, and if so, assesses the relevant sentences."

Key Insights Distilled From

VerAs: Verify then Assess STEM Lab Reports

by Berk Atil,Ma... at arxiv.org 04-26-2024

https://arxiv.org/pdf/2402.05224.pdf

Deeper Inquiries

How could VerAs be extended to provide more detailed feedback to students beyond the rubric scores, such as targeted suggestions for improvement?

VerAs could be extended to provide more detailed feedback to students by incorporating natural language generation (NLG) techniques. After assessing the lab reports or essays based on the rubric dimensions, VerAs could generate personalized feedback for each student. This feedback could include specific suggestions for improvement based on the strengths and weaknesses identified in the student's writing. By analyzing the content of the report and comparing it to the expected criteria, VerAs could provide targeted feedback on areas such as clarity of explanation, use of evidence, organization of ideas, and overall coherence. Additionally, VerAs could suggest resources or examples for further study to help students enhance their scientific writing skills.

What other types of STEM writing, beyond lab reports and essays, could benefit from the VerAs approach, and how would the architecture need to be adapted?

The VerAs approach could be beneficial for assessing various types of STEM writing, such as research papers, project reports, technical documentation, and scientific articles. To adapt the architecture for different types of STEM writing, the verifier and grader modules could be customized to align with the specific requirements of each genre. For research papers, the system could focus on evaluating the clarity of hypotheses, methodology descriptions, results interpretation, and conclusion coherence. In technical documentation, the emphasis could be on the accuracy of instructions, clarity of diagrams, and consistency in terminology. By tailoring the verifier and grader components to the unique characteristics of each type of STEM writing, VerAs could provide valuable feedback across a range of scientific communication formats.

Given the importance of science communication skills, how could the insights from this work on automated assessment be leveraged to improve the teaching of scientific writing more broadly?

The insights from automated assessment using VerAs could be leveraged to enhance the teaching of scientific writing by providing educators with valuable data and analytics on student performance. By analyzing the patterns in student writing, identifying common areas of strength and weakness, and tracking progress over time, educators can gain a deeper understanding of their students' needs. This data-driven approach can inform instructional strategies, curriculum development, and targeted interventions to support students in improving their science communication skills. Additionally, the automated feedback generated by VerAs can supplement manual feedback from instructors, offering students a comprehensive and consistent evaluation of their writing. By integrating automated assessment tools like VerAs into the teaching of scientific writing, educators can create a more efficient and effective learning environment that fosters the development of critical communication skills in STEM disciplines.