Fostering Design Questioning Skills Through Role-playing Interactions with Large Language Models

Core Concepts

Leveraging role-playing interactions with Large Language Models can help novice design students improve their ability to independently identify and clarify design problems.

Abstract

This study explores the use of Large Language Model (LLM)-powered Conversational Agents (CAs) to foster the questioning skills of novice design students for identifying design problems. The researchers developed a research probe that allows students to role-play as instructors and interact with an LLM-powered CA, which represents a novice design student.
The key findings include:

Role-playing interactions with the LLM-powered CA helped break down barriers for students to ask critical questions, as the non-judgmental nature of the CA allowed them to be more uninhibited.

Students used the LLM's responses as a quality indicator to develop more specific and higher-order questions, drawing from their previous experiences of receiving feedback from instructors.

However, students also faced challenges, such as repeatedly asking the same type of questions and becoming over-reliant on the LLM's responses, which hindered their independent critical thinking.

Integrating the insights from the questioning dialogues into a coherent design problem statement was also challenging for students, as they struggled to associate the argumentation with the design problem context.

The researchers discuss design implications for LLM-powered CAs to better support design students in improving their questioning skills, including adjusting the LLM's knowledge boundaries, encouraging real-time reflection on questions, and guiding students to ask questions that align with the nature of design.

Stats

"Identifying a design problem is a crucial step for deriving plausible solutions within the design's inherent ambiguity and iterative nature."
"Novice design students tend to rely on the instructor's feedback or accept it passively, which may hinder their ability to define design problems independently."
"Students asked a total of 172 inputs, with 53 Low-level Questions (LLQs), 43 Deep Reasoning Questions (DRQs), and 60 Generative Design Questions (GDQs)."

Quotes

"Since LLM is not a real person, I tried to ask questions as critically as possible without feeling guilty and with the intention of making it cry."
"At first, I found it challenging to ask questions, often asking broad ones, which led to the GPT providing long-winded and predictable responses. Then, I suddenly remembered the questions my professor was asking."
"This answer seems like you have simply arranged the former answer. I don't like it. Try to focus on the common grounds of the purposes and find the key user needs."

Key Insights Distilled From

Identify Design Problems Through Questioning: Exploring Role-playing Interactions with Large Language Models to Foster Design Questioning Skills

by Hyunseung Li... at arxiv.org 09-12-2024

https://arxiv.org/pdf/2409.07178.pdf

Deeper Inquiries

How can LLM-powered CAs be designed to strike a balance between providing helpful responses and encouraging independent critical thinking in design students?

To effectively balance the provision of helpful responses and the encouragement of independent critical thinking in design students, LLM-powered Conversational Agents (CAs) should be designed with specific features that promote active engagement and critical questioning. One approach is to implement a tiered response system where the LLM first provides a basic answer to a student's query, followed by prompts that encourage deeper inquiry. For instance, after answering a question, the LLM could ask, "What do you think about this solution?" or "Can you identify any potential drawbacks?" This method not only provides immediate assistance but also nudges students to reflect critically on the information presented.
Additionally, the LLM's knowledge boundaries should be carefully calibrated to ensure that it does not overwhelm students with excessive information. By limiting the depth of responses based on the student's level of expertise, the LLM can encourage students to seek clarification and ask follow-up questions, fostering a more interactive dialogue. Furthermore, integrating real-time feedback mechanisms that allow students to assess the quality of their questions can enhance their questioning skills. For example, the LLM could provide feedback on whether a question is too broad or if it could be refined for better clarity, thus promoting a learning environment that values critical thinking and self-assessment.

What other educational strategies or techniques could be combined with LLM-powered role-playing interactions to further enhance design students' questioning skills?

To further enhance design students' questioning skills in conjunction with LLM-powered role-playing interactions, several educational strategies can be integrated. One effective technique is Problem-Based Learning (PBL), which encourages students to engage with real-world problems and develop solutions collaboratively. By combining PBL with LLM interactions, students can use the LLM to simulate discussions around their design problems, allowing them to practice formulating questions in a context that mirrors real-life scenarios.
Peer review sessions can also be incorporated, where students present their design problems and the questions they have formulated to their peers. This collaborative approach not only fosters a sense of community but also allows students to receive diverse perspectives on their questioning strategies. The LLM can facilitate these sessions by providing prompts or guiding questions that help students refine their inquiries based on peer feedback.
Additionally, incorporating reflective practices, such as journaling or group discussions after LLM interactions, can help students critically analyze their questioning processes. By reflecting on what types of questions were effective and which were not, students can develop a deeper understanding of the art of questioning in design contexts. This combination of strategies can create a rich educational environment that nurtures critical thinking and enhances the overall learning experience.

How might the insights from this study on fostering design questioning skills be applied to other creative domains beyond design, such as entrepreneurship or scientific research?

The insights gained from fostering design questioning skills through LLM-powered interactions can be effectively applied to other creative domains, such as entrepreneurship and scientific research. In entrepreneurship, the ability to ask critical questions is essential for identifying market needs, evaluating business models, and refining product ideas. By utilizing LLM-powered CAs, aspiring entrepreneurs can engage in role-playing scenarios where they simulate investor pitches or customer feedback sessions. This practice can help them develop the questioning skills necessary to navigate complex business environments and make informed decisions.
In the realm of scientific research, the iterative process of hypothesis formulation and testing relies heavily on critical questioning. LLM-powered CAs can assist researchers in refining their research questions, exploring alternative hypotheses, and evaluating the implications of their findings. By engaging in role-playing interactions with the LLM, researchers can practice articulating their questions in a way that encourages deeper exploration and critical analysis of their work.
Moreover, the principles of questioning and critical thinking can be integrated into educational curricula across various disciplines. By emphasizing the importance of questioning in learning environments, educators can cultivate a culture of inquiry that transcends specific fields, empowering students to become more adept at navigating ambiguity and complexity in their respective domains. This holistic approach to education can ultimately lead to more innovative and effective problem-solving across creative fields.