Exploratory Study on Computing Students' Use of Large Language Models in Software Engineering Projects

Core Concepts

Students utilize Large Language Models (LLMs) for coding support, writing assistance, idea generation, and project management in software engineering projects, impacting their learning experiences.

Abstract

The study explores how upper-level computing students at Purdue University used Large Language Models (LLMs) like ChatGPT and Copilot in a semester-long software engineering project. The research aimed to understand students' experiences and perceptions of LLMs. The study collected data through interviews, revealing insights into how students integrated LLMs into coursework and how it influenced their learning. Key themes included LLM usage for coding support, writing assistance, idea generation, and project management. Students expressed concerns about knowledge retention, over-reliance on LLMs, and the need for prerequisite knowledge. The study highlighted the importance of responsible LLM usage and the potential impact on learning outcomes. Background & Related Work LLMs are transforming software engineering education. Students use LLMs for technical tasks, writing support, idea generation, and project management. Concerns exist about knowledge retention, over-reliance, and prerequisite knowledge. Future studies should explore LLM usage in lower-level courses and its impact on learning outcomes.

Stats

"Large Language Models (LLMs) are machine learning models trained on vast amounts of data." "LLMs have diverse applications in academia and industry, including code generation and explanation." "Students found LLMs useful for technical tasks like code synthesis and professional tasks like writing support."

Quotes

"LLMs are transforming the discipline of software engineering." "Using LLMs allows students to research and gather intelligence more efficiently." "Students expressed concerns about becoming too reliant on LLMs to complete their projects."

Key Insights Distilled From

An Exploratory Study on Upper-Level Computing Students' Use of Large Language Models as Tools in a Semester-Long Project

by Ben Arie Tan... at arxiv.org 03-28-2024

https://arxiv.org/pdf/2403.18679.pdf

Deeper Inquiries

How can LLMs be integrated responsibly into lower-level computer engineering courses?

Integrating Large Language Models (LLMs) into lower-level computer engineering courses requires a thoughtful approach to ensure responsible usage by students. Here are some strategies to consider: Introduction to Basics: Before introducing LLMs, ensure that students have a solid foundation in programming concepts and languages. This will help them understand the outputs generated by LLMs and prevent blind reliance on the technology. Guided Exercises: Provide structured exercises where students can interact with LLMs under supervision. This can help them understand the capabilities and limitations of LLMs while guiding them on how to use the technology effectively. Ethical Guidelines: Educate students on the ethical use of LLMs, including issues related to plagiarism, bias, and responsible information sourcing. Emphasize the importance of critical thinking and independent problem-solving alongside LLM usage. Collaborative Projects: Encourage collaborative projects where students can work together to leverage LLMs for research, idea generation, and problem-solving. This promotes teamwork and ensures that students are not overly dependent on LLMs for individual tasks. Feedback and Reflection: Incorporate opportunities for students to reflect on their use of LLMs in assignments and projects. Encourage them to analyze the impact of LLMs on their learning process and outcomes, fostering a deeper understanding of when and how to use LLMs effectively.

How can student interactions with LLMs be leveraged to provide feedback for course improvement?

Student interactions with Large Language Models (LLMs) can offer valuable insights for course improvement. Here are some strategies to leverage these interactions for feedback: Surveys and Interviews: Conduct surveys or interviews with students to gather their experiences and feedback on using LLMs in the course. Ask about the effectiveness of LLMs in aiding their learning, any challenges faced, and suggestions for improvement. Usage Analytics: Track student interactions with LLMs during assignments and projects. Analyze the patterns of usage, types of queries posed to LLMs, and the outcomes generated. This data can provide valuable insights into how students are utilizing LLMs in their coursework. Focus Groups: Organize focus group discussions with students who have extensively used LLMs. Encourage open dialogue about their experiences, preferences, and areas where they feel LLMs could be better integrated into the curriculum. Iterative Course Design: Use student feedback on LLM usage to iteratively design course materials and assignments. Incorporate suggestions for enhancing LLM-related activities, providing additional support where needed, and addressing any concerns raised by students. Faculty Reflection: Encourage faculty members to reflect on student feedback and adapt their teaching strategies accordingly. This continuous feedback loop ensures that the course remains responsive to student needs and evolving technologies like LLMs.